REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson 
Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE 

30.Jun.99 _ 


4.  TITLE  AND  SUBTITLE 

REDUCING  DOD  PRODUCT  DEVELOPMENT  TIME:  THE  ROLE  OF  THE 
SCHEDULE  DEVELOPMENT  PROCESS 


6.  AUTHOR(S) 

CAPT  MCNUTT  ROSS  T 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 


REPORT  TYPE  AND  DATES  COVERED 

DISSERTATION 


5.  FUNDING  NUMBERS 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 
THE  DEPARTMENT  OF  THE  AIR  FORCE 
AFIT/CIA,  BLDG  125 
2950  P  STREET 
WPAFB  OH  45433 


10.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

FY99-185 


12a.  DISTRIBUTION  AVAILABILITY  STATEMENT 

Unlimited  distribution 

In  Accordance  With  AFI  35-205/ AFIT  Sup  1 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  ( Maximum  200  words } 


0<Z0T  0  20 


14.  SUBJECT  TERMS 

15.  NUMBER  OF  PAGES 

543 

16.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION  18.  SECURITY  CLA! 
OF  REPORT  OF  THIS  PAGE 


OF  ABSTRACT 


ABSTRACT 


jyXIC  QUALITY  INSPECTED  8 


Standard  Form  298  (Rev.  2-89)  (EG) 

Prescribed  by  ANSI  Std.  239.18 

Designed  using  Perform  Pro,  WHS/DIOR,  Oct  94 


Reducing  DoD  Product  Development  Time: 
The  Role  of  the  Schedule  Development  Process 

by 


Ross  McNutt 

B.S.  United  States  Air  Force  Academy,  1987 
M.S.  Technology  and  Policy,  MIT,  1992 
M.S.  Aeronautics  and  Astronautics,  MIT,  1992 


Submitted  to  the  Department  of  Aeronautics  and  Astronautics 
in  partial  fulfillment  of  the  requirements  for  the  degree  of 

DOCTOR  OF  PHILOSOPHY 
at  the 

MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY 
December  1998 

©  1998  Ross  T.  McNutt,  All  Rights  Reserved. 


The  author  hereby  grants  to  MIT  permission  to  reproduce  and  distribute  publicly  paper  and 
electronic  copies  of  this  thesis  document  in  whole  or  in  part,  and  to  grant  others  the  right  to  do  so. 


Author., 


Technology,  Management,  and  Policy  Program 
Program  jn  Science  and  Technology  for  International  Security 

December  1998 


Certified  by . 

Director,  Program^ 
Certified  by . 


'■AH . . . 

KostaTsipis 

'in  Scienc|  and  Technology  for  International  Security 


John  Deyst 

Professor,  Department  of  Aeronautics  and  Astronautics 


Certified  by. 


Earll  Murman 
Professor,  Department  of  Aeronautics  and  Astronautics 


Accepted  by. 


Chairman,  Technology, 


Gchard  de  Neufville 
igement,  and  Policy  Program 


Abstract 


According  to  the  Packard  Commission,  “Unreasonably  long  acquisition  cycles  —  ten  to 
fifteen  years  for  major  weapon  systems  is  a  central  problem  from  which  most  other  acquisition 
Problems  stem.”  Since  the  commission  issued  its  report  in  1986,  the  time  required  to  develop 
new  military  systems  has  only  grown.  This  research  and  its  recommendations  are  intended  to 
identify  and  eliminate  the  causes  of  those  long  development  times  for  military  systems.  This 
report  addresses  a  key  factor  in  determining  the  development  time  for  military  projects:  the 
project’s  initial  schedule.  Part  1  outlines  the  current  situation,  previous  efforts  to  reduce 
development  time,  and  experiences  with  cutting  development  time  in  the  commercial  sector.  It 
also  documents  the  military  product  development  process.  Part  2  identifies  a  key  area-the 
schedule  development  process,  and  its  impact  on  development  time--for  in-depth  research. 
Through  understanding  what  is  driving  the  initial  project  schedule  and  the  impact  of  the  initial 
project  schedule  on  the  eventual  development  time,  the  author  identifies  key  drivers  of 
development  time.  Part  3  presents  the  results  of  three  surveys  and  analyzes  the  processes  used  to 
develop  a  project’s  initial  schedule,  the  process  used  to  develop  a  contracted  schedule,  and  the 
impact  of  these  schedules  on  actual  development  time.  Part  4  presents  observations,  draws 
conclusions,  and  makes  specific  recommendations  for  remedial  action. 

The  key  barriers  to  reducing  development  time  for  military  systems  are  the  lack  of 
importance  placed  on  project  schedules;  the  lack  of  effective  schedule-based  information  and 
tools;  the  lack  of  schedule-based  incentives;  and  the  overriding  impact  of  the  funding-based 
limitations  on  defense  projects.  The  steps  necessary  to  establish  a  focus  on  reducing 
development  time  are:  1)  recognizing  the  impact  of  development  time,  2)  providing  the  necessary 
information  for  decision  makers,  3)  providing  proper  incentives  at  each  organizational  level,  and 
finally  providing  a  structure  to  effectively  manage  the  set  of  all  development  projects  to  ensure 
that  each  project  can  be  funded  based  on  its  development  related  requirements. 

Implementing  the  recommendations  and  focusing  on  reducing  development  time  will 
force  other  changes  in  the  acquisition  process.  The  focus  on  reducing  the  time  to  develop  and 
field  systems  will  drive  the  acquisition  system  to  better  meet  the  needs  of  our  warfighters,  more 
rapidly,  and  at  lower  cost.  Better,  Faster,  and  Cheaper.  Even  more  importantly,  shortening 
development  times  is  critical  to  develop  and  produce  with  limited  resources,  the  right  weapons  at 
the  right  time  to  deter  or  to  defeat  any  potential  enemy  at  any  time  with  the  minimum  cost  to  our 
warfighters. 


3 


Prologue  and  Overview 


The  U.S.  Department  of  Defense  spearheads  the  world’s  largest  product  development 
operation.  In  1997  alone  it  spent  $32.5  billion  developing  new  or  improved  military  systems.  In 
the  past  20  years  DoD  has  spent  the  1998  equivalent  of  $732.5  billion  on  researching, 
developing,  testing,  and  evaluating  new  systems.  Through  these  critical  activities,  the  DoD 
creates  the  equipment  its  forces  need  to  fight  and  win.  An  effective  product  development  system 
should  ensure  that  those  forces  have  the  right  weapons  at  the  right  time  in  the  right  quantity  to 
deter  any  potential  enemy. 

Product  development  activities  determine  which  systems  will  be  built,  what  capabilities 
they  will  have,  how  reliable  they  will  be,  when  they  will  be  available,  and  how  much  they  will 
cost.  Product  development  also  influences  how  many  systems  are  fielded,  how  the  systems  can 
be  employed,  and  how  they  will  be  maintained.  Improving  the  performance  of  the  DoD  product 
development  system  could  result  in  more  effective  weapon  systems,  acquired  at  lower  cost,  in  a 
shorter  time. 

Over  the  last  10  years,  companies  in  a  wide  array  of  industries  have  made  dramatic 
improvements  in  their  approaches  to  creating  new  products,  largely  by  focusing  on  cutting 
development  time.  Companies  with  this  focus  often  achieve  50%  to  70%  reductions  by  forcing 
continuous  improvement  across  their  entire  organization.  The  result  is  higher-quality  products 
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that  are  easier  to  manufacture  that  are  ready  for  market  faster,  and  that  can  be  sold  for  a  higher 
profit.  These  companies  have  also  dramatically  increased  the  number  and  variety  of  products 
they  manufacture.  Development  time  is  a  key  component  of  a  company’s  ability  to  meet 
customers’  changing  needs.  Many  companies  and  industry  leaders  see  shorter  development 
times  and  more  effective  product  development  as  the  key  to  long-term  competitive  advantage. 

Several  previous  DoD  efforts  to  reduce  development  time  have  proven  ineffective.  In 
1983  Dr.  Jacques  Gansler  of  led  the  Air  Force  Affordable  Acquisition  Approach  study,  which 
examined  109  DoD  programs  to  answer  the  question,  “Does  it  really  take  longer  and  cost  more  to 
develop  systems  now  or  is  it  just  perception?”  The  study  group  found  that  development  time  had 
indeed  increased  significantly  since  the  1950s  and  made  a  series  of  specific  recommendations  to 
reduce  them.  In  1986,  the  Packard  Commission’s  Acquisition  Task  Force,  headed  by  Dr. 
William  Perry,  stated  in  its  Formula  for  Action  that  excessively  long  acquisition  cycles  are  “a 
central  problem  from  which  most  other  acquisition  problems  stem,”1  concluding  that  it  was 
“possible  to  cut  this  cycle  in  half.”2  The  Packard  Commission  then  made  its  own  series  of 
recommendations.  President  Reagan  directed  DoD  to  implement  the  Packard  Commission 
recommendations,  and  Congress  legislated  recommended  organizational  changes.  Yet  since 
1986  the  development  time  for  major  defense  systems  has  continued  to  grow:  major  new  systems 
fielded  during  the  1990s  have  required  an  average  of  1 1  years  from  the  decision  to  proceed  to 
initial  operational  capability.  The  time  from  decision  to  proceed  to  delivery  of  the  first 
production  item  accounts  for  1 12  of  the  average  132  months. 

Despite  continuing  growth  in  development  times,  recent  acquisition  reform  efforts  have 
focused  almost  solely  on  reducing  costs.  While  conducting  this  research,  the  author  has 
encountered  many  people  within  DoD  and  the  military  services  who  do  not  believe  that  reducing 
development  time  is  important.  One  senior  DoD  acquisition  reform  leader  stated  bluntly  that  she 
was  not  interested  in  such  an  effort  unless  it  would  cut  costs.  Survey  results  obtained  as  part  of 


1  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “A  Formula  For  Action:  A  Report  to  the  President 
on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.  Pg.  8. 

2  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “A  Formula  For  Action:  A  Report  to  the  President 
on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.  Pg.  15. 
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this  research  confirm  the  low  priority  placed  on  reducing  development  time  by  the  acquisition 
community. 

The  objective  of  this  research  is  to  improve  the  DoD  acquisition  process  to  allow  better 
equipment  to  reach  warfighters  more  quickly  and  at  lower  cost.  Cutting  development  times  is 
also  key  to  developing  systems  that  respond  effectively  to  changing  military  needs  and  take 
advantage  of  new  technologies.  An  overall  shift  in  strategy  away  from  developing  a  wide  array 
of  systems  to  counter  all  potential  threats  toward  one  more  responsive  to  specific  threats  will 
ensure  that  the  right  weapons  are  ready  when  they  are  needed. 

Cutting  development  cycles  will  also  extend  the  time  that  forces  maintain  their  technical 
advantage.  If  a  system  is  quickly  compromised  by  countermeasures,  its  long-term  worth  is 
questionable.  Thus  the  U.S.  Air  Force  now  builds  systems  “just-in-case”  as  insurance  against  a 
“pop  up”  threat.  That  is  one  argument  for  the  F-22  and  the  Joint  Strike  Fighter,  both  of  which 
are  intended  to  counter  threats  that  may  develop  over  the  next  30  years.  Indeed,  while  70%  of 
the  projects  surveyed  were  designed  to  meet  current  operational  deficiencies,  fully  30%  were 
intended  to  meet  projected  needs. 

The  net  effect  of  such  a  strategy  is  that  some  systems  are  often  developed  to  counter 
threats  that  never  materialize.  What’s  more,  when  systems  are  finally  fielded  they  often  must 
perform  a  dramatically  different  mission  than  they  were  designed  for.  This  produces  the  worst 
type  of  inefficiency:  developing  and  producing  the  wrong  weapon  at  the  wrong  time  with 
obsolete  technology,  or  one  that  does  not  meet  the  current  need  when  fielded.  The  B-2,  designed 
as  a  long-range  nuclear  bomber,  and  the  Milstar,  conceived  as  a  strategic  communication  system 
for  the  second  stage  of  a  nuclear  exchange,  are  visible  examples  of  systems  designed  to  meet 
missions  that  never  materialized  or  disappeared  before  they  were  completed.  The  Milstar  has 
been  used  instead  as  a  tactical  terminal  something  which  its  low  data  rate  makes  it  ill  suited  in 
today’s  digital  environment  and  large  demand  for  information.  The  “just-in-case”  strategy  also 
forces  the  US  to  spread  its  resources  over  many  systems,  resulting  in  even  longer  development 
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To  change  the  basic  development  philosophy  to  a  “just-in-time”  strategy,  where  systems 
are  fielded  as  they  are  needed  to  maintain  technological  superiority,  the  US  needs  to  be  able  to 
both  develop  and  field  a  sufficient  number  of  units  quickly.  Today  the  US  takes  10  to  20  years 
to  develop  and  field  major  new  weapon  systems,  which  are  expected  to  be  in  operation  for  20  to 
25  years.  That  approach  forces  the  US  to  project  threats  40  years  into  the  future  -  a  nearly 
impossible  feat.  The  requirements  that  the  F-22  and  other  planes  retain  technological  superiority 
in  the  year  2020  and  beyond  is  what  pushes  their  demanding  technology  performance 
requirements  and  drives  much  of  the  cost  of  military  systems.  In  general,  the  recent  response  to 
long  cycle  times  has  been  to  establish  a  long  term  planning  organization  in  the  Pentagon  to 
project  the  required  force  structure,  the  weapons,  and  the  technologies  required  a  full  25  to  40 
years  in  advance.  This  may  not  be  that  bad  except  that  the  effect  of  these  plans  is  to  lock  in  the 
planned  future  systems  and  related  schedules  long  before  it  is  possible  to  know  the  actual  threat. 
Once  made  these  plans  and  schedules  prove  very  difficult  and  costly  to  change.  The  system  is 
most  closely  comparable  to  the  central  planning  function  of  the  former  Communist  systems  with 
little  free  market  or  entrepreneurial  forces  at  work.  Bureaucratic  processes  dominate  the 
planning  processes  and  one  result  of  this  process,  as  shown  by  the  research,  is  long  development 
times. 


One  effect  of  long  development  times  for  weapon  systems  is  a  large  number  of  projects 
in  development  at  once.  This  is  the  same  problem  found  in  many  commercial  firms  that  are 
experiencing  product  development  problems.  Too  many  products  and  too  few  resources  to 
execute  them  effectively  and  efficiently  result  in  long  development  times,  significant  schedule 
slips,  and  cost  overruns.  As  will  be  shown,  exactly  what  is  happening  in  Air  Force  development 
efforts.  It  also  forces  large  technology  step  sizes  to  ensure  they  are  technologically  superior 
when  they  are  eventually  fielded,  which  only  exacerbates  the  problem. 

As  an  alternative  to  the  “just-in-case”  strategy,  one  could  go  to  a  “just-in-time”  strategy, 
where  systems  would  be  quickly  developed  to  counter  actual  identified  emerging  threats. 
Resources  could  be  more  effectively  targeted  to  counter  the  real  threats  instead  of  being  spread 
over  many  systems  to  counter  many  potential  emerging  threats.  This  could  allow  for  a  more 
targeted  approach  in  deciding  which  systems  are  needed  and  should  be  developed. 
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Having  an  effective  and  quick  development  process  would  allow  a  rapid  response  to  a 
“pop  up”  threat  that  may  be  based  on  a  new  technology  which  would  limit  the  duration  and  depth 
of  any  military  exposure.  Without  an  efficient  and  responsive  development  process,  the  US  is 
vulnerable  to  unpredicted  emerging  systems  based  on  new  technologies  or  novel  combinations  of 
existing  technologies. 

A  historical  point  is  that  this  “just-in-time”  approach  is  not  new.  It  has  been  the 
traditional  US  military  strategy  from  the  Revolution  through  the  Second  World  War.  Only 
during  the  Cold  War  did  we  feel  the  need  to  have  long  term  technological  superiority  over  all 
enemies.  During  W.W.n,  the  US  developed  better  systems  quickly  and  produced  them 
massively.  The  speed  and  potential  effects  of  the  nuclear  war  and  our  lack  of  preparation  for  the 
Korean  War  made  the  US  fear  being  caught  unprepared.  This  developed  into  the  idea  of  the 
“come  as  you  are  war”  which  would  be  fought  in  Europe  and  the  ultimate  winner  would  be 
determined  in  a  matter  of  weeks  before  any  production  could  effect  the  outcome.  The  fear  of  this 
type  of  total  lightning  war  has  been  eliminated  by  the  changes  in  Russia  and  the  lessening  of  the 
perceived  nuclear  threat.  Smaller  operations  will  still  be  on  a  “come  as  you  are”  basis  requiring 
significant,  but  not  massive,  inventories  of  the  highest  technology  weapons. 

This  fast  development  strategy  may  rely  on  a  pre-developed  set  of  critical  technologies 
and  subsystem  components  that  are  developed  to  the  point  where  they  can  be  assembled  quickly, 
tested,  and  then  produced  as  needed  en  masse.  Such  key  technologies  would  be  in  the  areas  of 
sensors,  signal  and  computer  processors,  communication  systems,  warheads,  autonomous 
control,  and  navigation.  The  “develop  and  test  but  not  produce”  strategy  would  keep  the  US 
aerospace  industry’s  product  development  capability  in  shape  to  ensure  that  it  could  be  called 
upon  when  needed.  It  would  facilitate  multiple  iteration  of  the  designs,  allowing  for  continual 
improvements  without  the  cost  of  modifications.  This  strategy  is  similar  to  the  design,  build,  test 
policies  or  the  “Silver  Bullet”  ideas  that  surfaced  during  the  early  1990’s.  Those  were  proposed 
more  because  of  a  lack  of  production  money  rather  than  a  shift  in  the  overall  military 
development  strategy  to  “just-in-time”.  The  “Silver  Bullet”  strategy  was  not  widely  accepted  for 
a  variety  of  political  reasons  which  may  have  changed  since  the  end  of  the  Cold  War  and  the 
acceptance  of  decreasing  defense  budgets. 
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Another  key  aspect  of  this  strategy  would  be  to  have  a  large  reserve  of  production 
capacity  that  would  be  available  in  time  of  an  emergency  or  an  emerging  threat.  The  central  and 
enabling  aspect  of  this  would  be  reliance  on  the  US  commercial  production  capacity  as  a  sort  of 
“Civil  Reserve  Industrial  Capacity,”  much  as  the  Air  Force  relies  on  the  Civil  Reserve  Air  Fleet 
to  augment  its  transport  aircraft  with  airline  aircraft.  This  requires  that  the  military  and 
commercial  production  capacities  be  merged.  The  merger  of  the  defense  and  commercial 
production  capacity  is  the  major  goal  of  the  previous  Defense  Acquisition  Executive,  Dr. 
Kaminski’s  efforts  to  remove  military  specification  and  standards  and  to  go  to  single  commercial 
standards  and  processes’.  Dr  Gansler,  the  current  Defense  Acquisition  Executive,  wrote  of 
similar  objectives  and  specified  specific  steps  to  begin  to  achieve  civil-military  integration  in  his 
book  Affording  Defense3 4  and  more  recently  in  a  June  3  letter  to  the  Service  Secretaries  on  the 
Single  Process  Initiative.5  Integration  of  the  defense  and  commercial  industries  allows  the  US  to 
maintain  the  defense  capacity  for  times  of  emergency  without  the  cost  of  constantly  producing 
defense  products  that  we  do  not  need  at  the  present  time.  This  also  allows  us  to  get  away  from 
the  “defense  industrial  complex”  requiring  constant  feeding  and  production  to  sustain  itself  as 
they  would  have  commercial  products  that  they  can  produce  to  maintain  their  capacity.  The  DoD 
would  pay  them  to  maintain  military  specific  equipment  and  capacities  and  maintain  the  ability 
to  rapidly  shift  to  produce  needed  military  systems.  Before  that  can  happen,  the  defense  industry 
must  either  become  lean  to  compete  in  the  commercial  market,  or  the  commercial  companies 
must  be  allowed  to  compete  in  the  defense  market  without  large  barriers.  If  one  reads  many  of 
the  Defense  Science  Board  studies  that  were  led  by  Dr.  Perry  and  Dr.  Kaminski  as  far  back  as  the 
1970’s,  the  emphasis  on  the  merging  of  the  defense  and  commercial  industry  is  a  central  theme 
and  for  this  very  reason.6  This  merging  of  production  lines  is  starting  to  occur  in  the  satellite, 


3  Perry,  William  “Secretary  of  Defense  Memorandum:  Common  Systems/ISO-9000/Expedited  Block  Changes”  Dec 
6, 1995. 

4  Gansler,  Jacques.  Affording  Defense.  Cambridge  MA:  MIT  Press,  1989  Pg.  239,  279-282. 

5  Gansler,  Jacques.  “USD(A&T)  Memorandum:  The  Single  Process  Initiative-A  Long  Term  Perspective.”  3  June 
1998. 

6  Department  of  Defense.  “Report  of  the  Defense  Science  Board:  1986  Summer  Study  on  Use  of  Commercial 
Components  in  Military  Equipment”  Office  of  the  Undersecretary  of  Defense  For  Acquisition.  Washington  D.C. 
January  1987. 
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electronics,  and  engine  industries,  but  to  date  it  has  not  occurred  in  the  aircraft  or  munitions 
areas. 

There  are  both  political  and  technology  reasons  that  make  the  change  from  “just-in-case”  to 
“just-in-time”  strategy  is  not  only  desirable  but  also  possible  in  today’s  environment  where  it  may 
not  have  been  possible  10  years  ago.  The  technology  based  reasons  are:  the  key  technologies  for 
military  systems  are  now  primarily  drawn  from  commercial  technology  developments;  the  rapid  pace 
of  commercial  technologies  has  overtaken  the  military  development  effort;  the  development  of  open 
system  architectures  from  commercial  computer  systems  is  making  its  way  into  military  systems 
(Open  Systems  Initiative);  the  quality  of  commercial  products  often  exceeds  the  military  standard 
equivalents;  and  the  acceptance  of  field  replaceable  instead  of  field  repairable  components  allows  the 
maintenance  concerns  to  be  reduced.  Some  of  the  political  changes  that  may  allow  for  this  type  of 
change  to  occur  now  are:  the  reduction  of  the  military  threat  and  a  period  of  clear  military  superiority 
(we  can  take  this  time  to  reorganize  without  exposing  ourselves  to  risk);  the  acceptance  of 
substantially  lower  long-term  defense  budgets;  the  diminished  threat  of  a  global  war  and  instead,  a 
focus  on  regional  conflicts;  and  a  recognition  on  the  part  of  industry  and  military  leaders  that  things 
can  and  must  change  if  we  are  to  provide  effective  weapons  for  our  forces  over  a  long  period. 

The  key  to  making  such  a  change  in  development  strategy  is  the  reduction  in  the  time  it 
takes  to  develop  and  produce  military  related  products.  The  demonstrated  capability  to  quickly 
develop  and  produce  weapons  is  the  key  enabling  capability  that  will  enable  such  a  change  to 
occur.  Not  only  is  reducing  the  product  development  time  a  key  to  improving  the  product 
development  process  but  it  is  also  key  to  changing  the  strategy  used  to  decide  which  systems  to 
develop  and  how  to  equip  our  forces. 

This  research  and  its  recommendations  are  intended  to  identify  and  eliminate  the  causes 
of  long  development  times  for  military  systems.  It  addresses  what  will  be  shown  to  be  a  key 
factor  in  determining  the  development  time  for  military  projects,  the  development  of  the  project’s 
initial  schedule.  This  report  is  organized  around  the  process  used  to  carry  out  the  study.  Part  1 
outlines  the  current  situation,  the  previous  efforts,  the  commercial  experience  associated  with 
development  time.  It  also  documents  the  product  development  process.  Chapter  1  identifies  the 
current  development  time  for  military  systems  and  found  that  they  have  consistently  increased 
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since  the  1970s.  It  identifies  some  of  the  impacts  of  development  time  on  military  capability, 
cost,  and  the  DoD  acquisition  system.  Chapter  1  also  identified  the  previous  significant  efforts  to 
shorten  development  times  such  as  the  Packard  Commission  and  other  more  contemporary 
efforts.  Chapter  2  looks  at  the  commercial  successes  at  improving  product  development 
capability  by  focusing  on  development  time.  It  provides  experiences  from  the  automobile 
industry  and  aerospace  industries  more  closely  related  to  the  defense  industry.  Chapter  3 
provides  a  descriptive  of  the  Air  Force  product  development  process  as  a  representative  example 
of  how  the  DoD  accomplishes  its  product  development  activities. 

Part  2  of  this  report  identifies  a  key  research  area,  the  schedule  development  process,  and 
its  impact  on  development  time,  for  in-depth  research.  Through  understanding  what  is  driving 
the  initial  project  schedule  and  the  impact  of  the  initial  project  schedule  on  the  eventual 
development  time,  the  author  hopes  to  identify  the  key  drivers  of  development  time.  Chapter  4 
identifies  the  specific  area  of  research  and  provides  the  framework  for  the  research.  Chapter  4 
also  reviews  the  previous  research  associated  with  project  schedules  associated  with  military 
projects.  Chapter  5  describes  the  research  method  selected  and  used  to  collect  the  required  data. 
It  describes  the  reasons  a  survey-based  method  was  selected  and  the  factors  that  were  considered 
in  developing  the  three  surveys  used  to  collect  project  level  data  at  the  Program  Offices,  the 
Pentagon,  and  the  Contractors.  Chapter  6  presents  the  demographics  of  the  3 17  survey 
respondents  to  show  that  it  is  a  representative  sample  of  the  current  development  efforts,  both 
large  and  small.  Chapter  7  describes  the  methods  used  to  analyze  and  illustrate  the  results  of  the 
surveys. 

Part  3  presents  the  results  of  the  three  surveys  and  the  analysis  of  the  processes  used  to 
develop  a  project’s  schedule  and  its  impact  on  actual  development  time.  Based  on  the  survey 
results,  Chapter  8  identifies  the  factors  involved  in  developing  a  project’s  initial  schedule.  It 
analyzes  user’s  schedule  desires,  the  project  objectives,  and  the  influences  on  schedule.  It 
identifies  the  schedule  information  and  tools  used,  and  the  organizations  involved  in  the 
development  of  a  project’s  initial  schedule.  Chapter  9  shows  the  impact  of  the  project’s  initial 
schedule  on  the  project’s  contracted  schedule.  It  shows  what  is  driving  the  length  of  the 
contractor’s  proposed  schedules  and  the  results  of  these  schedules.  Chapter  10  identifies  the 
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incentives  associated  with  the  length  of  project  schedules  for  the  development  of  a  project.  It 
describes  both  the  organizational  and  personal  incentives  at  the  Pentagon,  Program  Offices,  and 
Contractor  for  associated  with  project  schedules.  Chapter  1 1  then  documents  the  achieved 
schedules  and  compares  them  to  both  the  initial  project  schedules  and  estimates  of  the  project 
managers  of  the  minimum  required  time  to  develop  the  projects.  It  shows  the  impact  on  the 
overall  development  time  that  the  project’s  initial  schedule  and  the  various  factors  that  cause 
delays  in  projects  has  on  the  time  it  eventually  takes  to  develop  the  project. 

Based  on  the  survey  results,  Part  4  draws  conclusions,  presents  observations,  and  makes 
specific  recommendations  for  remedial  action.  Chapter  12  draws  specific  conclusions  on  the 
schedule  development  process  and  its  impact.  Chapter  13  places  these  conclusions  in  the  larger 
context  and  makes  specific  observations  about  the  entire  product  development  process.  Chapter 
14  makes  specific  recommendations  for  action  that  must  be  taken  to  improve  the  product 
development  process  and  in  particular  the  process  used  to  develop  the  initial  schedules  of  a 
development  project.  Chapter  14  then  revisits  the  larger  changes  in  development  strategies 
enabled  by  the  reduction  in  development  times  and  how  shorter  development  times  may  allow 
for  a  change  from  a  “just-in-case”  development  strategy  to  “just-in-time”  approach. 

The  ultimate  objective  of  this  research  is  to  reduce  development  time  in  order  to  meet  the 
ever-changing  needs  of  our  warfighters.  Reducing  development  time  is  the  key  to  improving  the 
development  process  to  allow  it  to  provide  higher  quality  products,  more  rapidly,  and  at  lower 
cost.  Current  long  development  times  for  military  systems  have  significant  negative  impacts. 
Many  commercial  firms  have  achieved  dramatic  improvements  in  their  product  development 
processes  by  focusing  on  reducing  development  time.  This  research  provides  a  detailed 
description  of  the  complex  processes  the  military  uses  to  select,  plan,  and  carry  out  development 
projects  and  the  impact  those  processes  have  on  the  time  it  takes  to  develop  and  field  these 
projects. 

This  research  identifies  key  barriers  to  reducing  development  time  and  makes  specific 
recommendations  on  how  to  remove  them.  The  key  barriers  to  reducing  development  time  for 
military  systems  are  the  lack  of  importance  placed  on  project  schedules;  the  lack  of  effective 
schedule-based  information  and  tools;  the  lack  of  schedule  based  incentives;  and  the  overriding 
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impact  of  the  funding-based  limitations  on  defense  projects.  The  steps  necessary  to  establish  a 
focus  on  reducing  development  time  are:  1)  recognizing  the  impact  of  development  time,  2) 
providing  the  necessary  information  for  decision  makers,  3)  providing  proper  incentives  at  each 
organizational  level,  and  finally  providing  a  structure  to  effectively  manage  the  set  of  all 
development  projects  to  ensure  that  each  project  can  be  funded  based  on  its  development  related 
requirements. 

The  implementation  of  the  recommendations  and  a  focus  on  reducing  development  time 
will  force  other  necessary  improvements  in  development  and  other  acquisition  processes.  The 
focus  on  reducing  the  time  to  develop  and  field  systems  will  drive  the  acquisition  system  to 
better  meet  the  needs  of  our  warfighters,  more  rapidly,  and  at  lower  cost.  Better,  Faster,  and 
Cheaper.  Even  more  importantly,  shortening  development  times  is  critical  to  develop  and 
produce  with  limited  resources,  the  right  weapons  at  the  right  time  to  deter  or  to  defeat  any 
potential  enemy  at  any  time  with  the  minimum  cost  to  our  warfighters.  The  recommendations  of 
this  effort  should  be  implemented  forthwith  to  begin  the  long  and  difficult  process  that  is 
required  to  shorten  the  development  times  and  increase  the  effectiveness  of  our  military  product 
development  system. 
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Parti 


Problem  Identification  and  Development 

Process  Overview 


Overview 

Part  1  of  this  thesis,  consisting  of  chapters  1  through  3,  describes  some  current  issues 
associated  with  the  Air  Force  product  development  process  and  the  problem  of  long  development 
cycles.  Chapter  1  identifies  the  nature  of  the  problems  that  lead  to  long  cycle  times  and  discusses 
past  and  current  efforts  to  address  them.  Chapter  2  describes  significant  advances  in  the 
commercial  sector  that  have  resulted  from  applying  lean  development  practices.  Chapter  3 
specifically  describes  Air  Force  product  development  processes  to  establish  a  frame  of  reference 
for  discussions  in  later  chapters. 
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Chapter  1 


Long  Development  Times  for 

Military  Systems _ _ 

The  U.S.  Department  of  Defense  oversees  the  world’s  largest  product  development 
operation.  Over  the  past  20  years,  DoD  has  spent  the  1998  equivalent  of  $732.5  billion  on 
researching,  developing,  testing,  and  evaluating  new  systems.  In  1997  alone  it  spent  $32.5 
billion  developing  new  or  improved  military  systems,  despite  a  30  percent  drop  in  annual 
development  dollars  since  1989. 

Today  development-related  activities  represent  42  percent  of  DoD’ s  total  procurement 
costs.  This  amount  does  not  include  the  tens  of  thousands  of  military  and  civilian  DoD 
employees  assigned  to  acquisition-related  positions,  nor  the  development-related  activities 
associated  with  operating  and  maintaining  existing  systems.  Overall,  the  money  going  to  R&D 
on  new  weapon  systems  represents  an  increasing  fraction  of  all  modernization  dollars,  and  this 
fraction  is  currently  at  an  all-time  high.  For  many  programs,  development  costs  significantly 
exceed  procurement  costs.  Development  costs  combined  with  the  rapid  rate  of  technological 
advances — plus  the  impact  on  operational  capabilities  and  costs — make  the  DoD  development 
process  a  central  issue  in  defense  acquisition. 
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Many  efforts  have  been  made  to  improve  the  military  acquisition  system  over  the  last 
several  decades.  The  most  influential  reform  initiative  was  the  Presidential  Blue  Ribbon 
Commission  led  by  David  Packard  in  1986.  The  Packard  Commission  identified  long 
development  time  as  the  key  problem  and  stated  in  its  conclusion: 


“Serious  result  of  this  management  environment  is  an  unreasonably  long  acquisition 
cycle-ten  to  fifteen  years  for  our  tnajor  weapon  systems.  This  is  a  central  problem  from 
which  most  other  acquisition  problems  stem: ...  it  leads  to  unnecessarily  high  costs  of 
development, . . .  obsolete  technology  . . .  and  aggravates  the  very  gold  plating  that  is  one 
of  its  causes.” 


Packard  Commission  Report7 8 

As  it  introduced  its  recommendations,  the  Packard  Commission  stated: 


“Acquisition  problems  have  been  with  us  for  several  decades,  and  are  becoming  more 
intractable  with  the  growing  adversarial  relationship  between  government  and  the  defense 
industry,  and  the  increasing  tendency  of  Congress  to  legislate  management  solutions.  In 
frustration,  many  have  come  to  accept  the  ten-to-fifteen  years  acquisition  cycle  as  normal, 
or  even  inevitable. 

“We  believe  that  it  is  possible  to  cut  this  cycle  in  half.” 

Packard  Commission  Report* 


Long  product  development  cycles  have  a  serious  impact  not  only  on  the  military’s 
acquisition  system  but  also  on  its  warfighting  capabilities.  DoD  leaders  have  long  complained 
that  it  takes  too  long  to  develop  and  field  new  systems  and  too  long  to  meet  the  needs  of 
warfighters.  In  1986,  the  Packard  Commission  reported: 


“The  Chairman  of  the  Joint  Chiefs  of  Staff  stated  that  the  most  important  way  technology 
could  enhance  our  military  capability  would  be  to  cut  the  acquisition  cycle  in  half.”9 


7  President’s  Blue  Ribbon  Commission  on  Defense  Management  (Packard  Commission).  “Formula  For  Action:  A 
Report  to  the  President  on  Defense  Acquisition”.  1986.  Pg.  8. 

8  Packard  Commission  “Formula  For  Action:  A  Report  to  the  President  on  Defense  Acquisition”.  1986.  Pg.  15. 

9  Packard  Commission  “Formula  For  Action:  A  Report  to  the  President  on  Defense  Acquisition”.  1986.  Pg.  35. 
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More  recently,  Secretary  of  Defense  William  Cohen  in  March  of  1997  stated: 

“ . . .  We  need  a  fast-paced  acquisition  system  that  can  seize  upon  new  technologies _ 

We  need  to  quickly  get  technology  to  the  warfighters’  hands  to  met  their  needs  while  . . . 
it  is  new  and  competitive.” 10 

General  Ronald  R.  Fogleman,  former  U.S.  Air  Force  Chief  of  Staff,  indicated  the  importance  of 
making  the  acquisition  system  more  responsive  to  warfighters’  needs: 


“It's  essential  that  we  make  our  acquisition  system  more  responsive  to  the  needs  of  the 
warfighter.  We’ve  engaged  our  partners  in  industry  to  help  us  improve  our  acquisition 
process  and  produce  more  combat  capability  for  the  dollars  we  invest. . . .  Working 
together,  we  can  provide  a  truly  responsive  acquisition  process  that  fields  the  capabilities 
required  to  underwrite  asymmetric  force  application  in  a  timely  and  cost-effective 
manner.”" 

The  Honorable  Paul  Kaminski,  undersecretary  of  defense  for  acquisition  and  technology,  put  it 
more  straightforwardly  when  he  told  a  Senate  Committee: 


“The  Department  of  Defense  cannot  afford  a  15-year  acquisition  cycle  time  when  the 
comparable  commercial  turnover  is  every  3  to  4  years.  The  issue  is  not  only  cost.  The 
lives  of  our  soldiers,  sailors,  marines,  and  airmen  may  depend  upon  shortened  acquisition 
cycle  times  as  well.  In  a  global  market,  everyone,  including  our  potential  adversaries, 
will  gain  increasing  access  to  the  same  commercial  technology  base.  The  military 
advantage  goes  to  the  nation  who  has  the  best  cycle  time  to  capture  technologies  that  are 
commercially  available;  incorporate  them  in  weapon  systems;  and  get  them  fielded 


,0  Cohen,  William  and  Kaminski,  Paul.  “DoD  Press  Conference”.  14  March  1997.  Reported  in  Program  Manager. 
Defense  Systems  Management  College  May  June  1997.  Pg.  15-18. 

11  Fogleman,  Gen.  Ronald  R.,  U.S.  Air  Force  Chief  of  Staff.  “Air  Power  and  the  American  Way  of  War”.  Presented 
at  the  Air  Force  Association  Air  Warfare  Symposium.  Orlando,  Florida.  Feb  15, 1996. 

12  Kaminski,  Paul  G.  “Statement  of  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Paul 
Kaminski,  before  the  Subcommittee  on  Defense  Technology,  Acquisition  and  Industrial  Base  of  the  Senate 
Committee  on  Armed  Services  on  Dual  Use  Technology”.  May  17,  1995. 
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A.  Acquisition  Response  Time 

The  acquisition  cycle  to  which  the  leaders  refer  can  also  be  described  as  the  acquisition 
response  time.  Acquisition  response  time  is  the  time  from  the  emergence  of  a  threat,  an 
operational  need,  or  a  new  technological  opportunity  to  the  delivery  of  enough  systems  to 
provide  operational  capability.  Acquisition  response  time  is  thus  a  measure  of  the 
responsiveness  of  the  entire  acquisition  system  to  meet  the  needs  of  warfighters.  This  time 
includes  recognition  time,  decision  time,  development  time,  and  production  time. 


Figure  1-1:  Components  of  Acquisition  Response  Time  —  Time  to  Meet  the  Customer’s  or 
User’s  Needs 


Recognition  time  begins  when  a  new  threat  emerges,  a  deficiency  in  the  ability  to  execute 
a  military  strategy  appears,  a  change  in  military  strategy  that  requires  new  systems  occurs,  or  a 
technological  opportunity  with  significant  military  application  emerges.  Recognition  time  ends 
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when  the  services  formally  recognize  the  need  for  a  new  system.  This  is  typically  done  through  a 
Mission  Needs  Statement. 

Decision  time  begins  when  the  need  is  recognized  and  ends  when  a  decision  is  made  to 
proceed  with  a  development  effort  to  fill  that  need.  During  this  period  due  leaders  must  decide 
which  needs  among  many  to  address,  and  which  opportunities  to  pursue.  During  this  time 
leaders  also  define  the  requirements  for  a  new  system,  determine  which  technology  to  use, 
allocate  the  resources,  and  create  a  development  plan.  Decision  time  ends  with  a  Milestone  I 
decision  or  project  approval. 

Development  time  begins  with  the  decision  to  start  the  project  and  ends  with  the  delivery 
of  the  first  production  item.  This  stage  includes  selecting  contractors,  refining  the  product’s 
design  and  the  process  used  to  make  it,  prototyping,  testing,  and  producing  the  first 
representative  system.  By  the  end  of  this  period,  the  vast  majority  of  product  and  process 
development  activities  are  complete.  Further  improvements  to  the  production  process  can  be 
attributed  to  the  learning  curve  during  production. 

Production  time  begins  with  completion  of  the  first  item  and  ends  when  enough  systems 
are  produced  and  fielded  to  provide  an  effective  operational  capability.  In  the  military  this  is 
marked  by  the  “required  asset  availability”  (RAA)  date,  or  by  the  declaration  of  “initial 
operational  capability”  (IOC).  Not  until  this  point  do  warfighters  consider  a  system  usable  for 
the  intended  mission. 

Determining  overall  acquisition  response  time  is  often  difficult  because  it  is  unclear 
exactly  when  an  opportunity  opens  or  a  threat  emerges.  Decisions  can  occur  quickly  if  an 
opportunity  or  threat  catches  a  leader's  attention,  but  potential  projects  can  also  languish  for  years 
within  the  modernization  planning  process,  never  managing  to  make  it  above  the  cutoff  line. 
Decision  makers  may  also  take  as  long  as  five  years  to  build  the  necessary  consensus  for  starting 
a  development  project. 
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B.  Development  Times  for  Military  Systems 

Development  time  is  easier  to  examine  and  analyze  since  it  has  a  more  defined  beginning 
and  end.  Development  time  for  all  major  defense  acquisition  programs13  averaged  106  months,  or 
nearly  9  years,  from  1965  to  1995,  as  shown  in  Figure  1-2  below.  Air  Force  ACAT II  projects— 
those  costing  between  $355  million  and  $140  million  for  development  averaged  66  months. 
Smaller  ACAT  HI  projects-those  costing  less  than  $140  million  for  development — averaged  51 
months. 


Total  Development  Time  (Months) 


Figure  1-2:  Distribution  of  Product  Development  Time  for  Major  Defense  Acquisition 
Systems  Since  196514 


13  Major  Defense  Acquisition  Programs  are  those  that  have  more  than  $355  million  in  projected  development  costs 
or  more  than  $1  billion  in  projected  production  costs. 

14  Jarvaise,  Drezner,  and  Norton.  “The  Defense  System  Cost  Performance  Database:  Cost  Growth  Analysis  Using 
Selected  Acquisition  Reports.”  Santa  Monica  CA:  Rand.  MR-625-OSD  Data  Current  as  of  December  1994. 
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Product  Development  Time  (Months) 

Figure  1-3:  Distribution  of  Development  Time  for  Smaller  Air  Force  Development  Projects 
(Current  ACAT II  and  ACAT  III  Projects)15 


Development  times  have  increased  since  the  Packard  Commission  report  was  issued  in 
1986.  Development  time  for  major  defense  projects,  based  on  the  date  of  the  first  delivered 
operational  item,  has  grown  from  97  months  in  the  first  half  of  the  1980s  to  108  months  in  the 
first  half  of  the  1990s.  Development  time  appears  to  be  headed  for  1 15  months  for  the  second 
half  of  1990,  and  to  over  12  months--more  than  10  years-after  2000,  as  shown  in  Figure  1-4. 
(No  future  data  are  available  for  smaller  projects.) 


The  figure  shows  the  times  from  program  start  (Milestone  1)  to  first  operational  delivery. 
The  figure  does  not  include  Pre-Milestone  1  activities,  which  can  last  up  to  5  years,  nor  does  it 
include  time  from  first  operational  delivery  through  delivery  of  enough  quantities  for  operational 
use. 

As  shown  in  Figure  1-5,  long  product  development  times  occur  across  all  types  of 
systems.  Of  particular  note  are  that  electronic  systems,  munitions  systems,  and  helicopters 
average  more  than  10  years  in  development. 


15  Data  provided  by  Secretary  of  the  Air  Force  (Acquisition)  SAF/AQXR.  Definitions  of  start  and  stop  points  used 
to  determine  development  time  could  not  be  verified. 
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Figure  1-4:  Average  Development  Time  for  Major  Defense  Acquisition  Programs  by  Year 
(Measured  from  Program  Start  (Milestone  1)  to  First  Operational  Delivery)16 


Figure  1-5:  Average  Time  from  Program  Initiation  to  First  Operational  Delivery  for  Major 
Defense  Acquisition  Systems.17 


16  Data  from  Rand  SAR  Database.  Jarvaise,  Drezner,  and  Norton.  The  Defense  System  Cost  Performance  Database: 
Cost  Growth  Analysis  Using  Selected  Acquisition  Reports.”  Santa  Monica  CA:  Rand.  MR-625-OSD  Data  Current 
as  of  December  1994.  Averaged  over  all  major  defense  acquisition  programs  with  available  data.  Note  the  70-74 
data  may  not  contain  some  long  running  program  data  that  was  not  included  in  the  SAR  database  as  they  preceded 
the  SAR  reporting  requirements. 
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Little  solace  can  be  taken  from  the  relative  speed  of  the  process  for  developing  military 
aircraft,  such  as  the  F-16,  KC-10,  and  A-10.  More  contemporary  efforts,  such  as  the  C-17  (150 
months),  F-22  (144  months),  and  T-45  (111  months),  average  significantly  longer  development 
times. 

Average  development  times  for  smaller  Air  Force  ACAT II  and  ACAT  III  projects 
(below  $355  million  in  development  costs)  are  shown  in  Figure  1-6  below. 


Figure  1-6:  Projected  Average  Development  Times  for  Current  Smaller  Programs  (ACAT 
II  and  ACAT  III  Projects) ,8 


17  Data  from  Jarvaise,  Drezner,  and  Norton.  The  Defense  System  Cost  Performance  Database:  Cost  Growth 
Analysis  Using  Selected  Acquisition  Reports.”  MR-625-OSD.  Santa  Monica  CA:  Rand.  Data  Current  as  of 
December  1994. 

18  Projected  development  time  provided  by  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQXR). 
Definition  of  start  and  stop  points  used  to  determine  development  time  could  not  be  verified.  Percent  of 
development  complete  also  could  not  be  determined. 
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For  new  programs  completed  in  the  1990s,  the  average  time  from  program  start  to  initial 
operational  capability— the  real  test— is  over  10  years.  This  average  is  available  from  a  recently 
developed  database  from  the  Office  of  the  Secretary  of  Defense.  This  database  is  based  on 
Selected  Acquisition  Reports  and  is  current  as  of  March  1998.'9 


Average 

Acquisition 

Time 

(Months) 


Year  of  Initial  Operational  Capability 


Figure  1-7:  Time  from  Program  Initiation  to  Initial  Operational  Capability  for  Major 
Defense  Acquisition  Programs  (from  OSD(A&T)  API  ACTS  database  of  215  Major 
Defense  Acquisition  Programs) 


The  scenario  feared  by  the  Packard  Commission — that  the  military  would  come  to  accept 
10-to-  15-year  acquisition  cycles — has  apparently  occurred  Systems  now  in  initial  stages  have 
been  planned  with  long  development  cycles.  The  newest  system,  the  Joint  Strike  Fighter,  which 
has  just  begun  its  demonstration  phase,  is  scheduled  to  be  operational  in  2008  after  12  years  of 
development.  The  AIM-9X  off-bore  sight  missile  used  in  close  air-to-air  combat  is  expected  to 
reach  initial  operational  capability  in  2003.  Nine  years  after  program  initiation  despite  the  fact 


19  Department  of  Defense  OSD(A&T)API.  DoD  Cycle  Time  Analysis  Tool  15  January  1998 
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that  a  prototype  was  demonstrated  as  early  as  1994,  and  despite  the  fact  that  the  weapon  fulfills  a 
strong  identified  need  and  that  it  has  significant  political  backing.20  Even  the  F/A-18  E/F,  an 
upgrade  of  an  existing  fighter,  is  scheduled  to  take  1 1  years  from  initial  contract  award  in  1991 
until  initial  operational  capability  in  2002. 

The  “nominal”  or  planned  times  for  proceeding  from  program  initiation  to  the  beginning 
of  production,  shown  in  Table  1-1  and  taken  from  a  Defense  Systems  Management  College 
chart,  range  from  6  to  13  years.  When  one  includes  the  production  process,  the  acquisition  cycle 
takes  8  to  21  years — potentially  one  year  longer  than  a  successful  military  career. 


Phase  0 

Concept  Exploration 

0-2  years 

Milestone  1 

Program  Initiation 

Phase  I 
Milestone  2 

Demonstration  and  Validation 

2-4  years 

Phase  II 
Milestone  3 

Engineering  and  Manufacturing  Development 

4-7  years 

Phase  III 

Production  and  Deployment 

2-8  years 

Phase  IV 

Support 

10-50  years 

6-13  years 
8-21  years 

Table  1-1  “Nominal”  or  Planned  Defense  Acquisition  Time  Scales21 


Development  Range 
Deployment  Range 


Exacerbating  long  planned  development  cycles,  only  half  of  DoD  development  programs 
meet  their  schedule.  Many  programs  are  delayed  for  years,  some  because  of  technical  problems, 
some  for  funding  reasons,  and  some  because  of  changing  requirements.  Electronic  systems  and 
munitions  systems,  for  example,  average  21  months’  delay.  Figure  1-8  below  shows  the  number 
of  major  defense  acquisition  programs  that  experience  schedule  slips.  Table  1-2  shows  the 
planned  and  actual  schedules  for  different  types  of  major  development  projects. 


20  Domheim,  M.  and  Hughes  D..  “U.S.  Intensifies  Efforts  to  Meet  Missile  Threat”.  Aviation  Week  and  Space 
Technology.  October  16, 1995.  Pg.  39  and  June  6, 1994.  Pg.  44. 

21  Defense  Systems  Management  College  Chart  -  Defense  Systems  Acquisition  Management  Process.  May  92  and 
97. 
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System  Type 

Planned 

(months) 

Achieved 

(months) 

Average  Slip 
(months) 

Number  of 
Programs 
in  Database 

All  Programs 
in  database 

90 

106 

14 

131 

Aircraft 

78 

83 

2 

23 

Ship 

98 

107 

8 

24 

Munitions 

87 

121 

26 

8 

Missile 

92 

105 

13 

25 

Space 

97 

109 

8 

6 

Electronic 

97 

121 

21 

29 

Vehicle 

76 

95 

14 

9 

Helicopter 

96 

131 

46 

5 

Other 

58 

68 

11 

2 

Table  1-2:  Months  from  Program  Initiation  to  First  Operational  Delivery  of  Major  Defense  Acquisition 
Programs” 


T otal  Program  Slip  (Months) 


Figure  1-8:  Distribution  of  Schedule  Slip  for  131  Major  Defense  Acquisition 
Programs  (mean  schedule  slip:  13.8  months)” 


”  Data  from  Jarvaise,  Drezner,  and  Norton.  The  Defense  System  Cost  Performance  Database:  Cost  Growth 
Analysis  Using  Selected  Acquisition  Reports.”  Santa  Monica  CA:  Rand.  MR-625-OSD.  Data  Current  as  of 
December  1994. 
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Year  of  Initial  Operational  Delivery 


Figure  1-9:  Average  Project  Slip  by  Date  of  Initial  Operational  Delivery  for  Major 
Defense  Acquisition  Program 

The  data  show  that  development  time,  including  the  slip  in  the  schedule,  averages  over 
106  months  for  major  defense  systems.  Recognition  time,  decision  time,  and  the  time  from  first 
production  item  to  delivery  of  enough  systems  for  effective  capability  only  add  to  these  lengthy 
periods.  The  last  of  these,  the  time  from  decision  to  proceed  to  initial  operating  capability  was 
the  primary  focus  of  the  Packard  Commission  faulting  the  excessively  long  acquisition  cycle  and 
their  recommendations  to  reforming  the  acquisition  system. 


C.  Impact  of  Long  Development  Times 

Long  development  times  impact  the  DoD  in  many  ways.  It  impacts  our  military  capability 
through  systems,  long  in  development,  not  being  ready  when  needed.  It  impacts  our  military 
capability  through  systems  not  meeting  the  current  need  when  fielded.  It  impacts  our  military 
capability  through  fielding  of  dated  technology  in  our  newest  systems.  It  impacts  our  ability  to 
quickly  respond  to  new  or  emerging  threats  or  to  respond  to  known  safety  issues.  Long 


23  Data  from  Jarvaise,  Drezner,  and  Norton.  The  Defense  System  Cost  Performance  Database:  Cost  Growth 
Analysis  Using  Selected  Acquisition  Reports.”  Santa  Monica  CA:  Rand.  MR-625-OSD.  Data  Current  as  of 
December  1994. 
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development  times  also  result  in  increased  cost  to  develop  and  sustainment  of  our  weapon 
systems.  Examples  of  each  type  of  impact  of  long  development  times  are  provided  in  appendix  2. 


C.l.  Systems  Not  Ready  When  Needed 

Desert  Storm  provided  a  unique  opportunity  to  identify  military  needs  in  a  wartime 
environment  and  determine  the  possible  impact  of  equipment  in  the  development  pipeline  had  it 
had  been  deployed  more  quickly.  Seven  systems  that  were  long  in  development  would  have 
mitigate  critical  needs  during  the  early  part  of  Desert  Shield  and  Desert  Storm.  These  systems 
were  the  C-17  strategic  and  tactical  heavy  lift  cargo  aircraft  (started  1980),  MILSTAR  survivable 
satellite  communications  (started  1981),  LANTRIN  Precision  Targeting  System  (started  1979), 
Joint  Tactical  Information  Distribution  System  (started  1974),  Global  Positioning  System 
(started  1973),  Advanced  Air-to-Air  Medium  Range  Missile  (started  1978),  Sensor  Fuse  Weapon 
wide-  area  anti-tank  capability  (started  1983).  Additional  details  are  provided  as  o  the  program 
circumstances  and  impact  in  Appendix  2. 

Many  other  major  defense  systems  under  development  for  at  least  five  years  were  not 
available  for  use  in  Desert  Shield  or  Desert  Storm.  Those  included  the  Stingray  Anti-Aircraft 
Missile,  the  V-22  Osprey,  the  AGM-130  Powered  Glide  Bomb,  the  Mark  XV  Identification 
Friend  or  Foe,  the  Army  Brilliant  Anti-Tank  Weapon,  the  Advanced  Apache  Long  Bow  and 
Hellfire  Missile  System,  the  Commanche  attack  helicopter,  the  AW  ACS  Block  30-35  upgrade 
program,  the  F-22  Air  Superior  Fighter,  and  the  B-2  strategic  bomber.  All  these  systems  were 
started  based  on  an  identified  need.  Many  of  these  systems  have  still  not  reached  operational 
status  7  years  later. 

A  few  systems  were  rushed  through  development  and  made  available  to  troops  during  the 
six  months  prior  to  Desert  Storm.  The  Joint  STARS  surveillance  plane  was  pushed  into  service 
and  provided  critical  observations  of  Iraqi  troop  movements.  A  bomb  system,  the  Bunker 
Buster,  was  developed  and  fielded  in  29  days.  This  rapid  action  demonstrated  that  the 
acquisition  system  can  move  quickly  to  meet  the  needs  of  warfighters  when  they  are  seen  as 
essential.  But  such  efforts  are  the  exception  rather  than  the  rule. 
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C.2.  Systems  Not  Meeting  Current  Needs  When  Fielded 

With  the  average  development  time  for  a  new  major  defense  system  approaching  10 
years,  the  need  for  and  requirements  of  any  system  in  development  may  dramatically  change. 
Many  of  the  systems  now  in  the  pipeline  are  based  on  the  threat  and  political  environment  that 
existed  before  the  dissolution  of  the  Soviet  Union  and  the  Gulf  War.  Of  26  current  major 
development  programs  in  the  Rand  database  and  due  for  completion  between  1995  and  1999, 
only  6  were  started  following  the  end  of  the  Cold  War.  Twelve  began  during  the  early  1980s, 
when  the  U.S.  faced  a  radically  different  environment.  One  result  is  that  systems  often  do  not 
adequately  meet  warfighters  current  needs  when  fielded. 

C.3.  New  Systems  Fielded  with  Dated  Technology 

In  the  time  now  required  to  develop  and  field  a  new  military  system,  technology  is  no 
longer  state  of  the  art,  and  in  some  cases  it  is  obsolete  and  out  of  production.  Ten  year  ago,  the 
Packard  Commission  stated  that  long  development  cycles  “lead  to  obsolete  technology  in  our 
fielded  equipment.  We  forfeit  our  five-year  technological  lead  by  the  time  it  takes  us  to  get  our 
systems  from  the  laboratory  into  the  field.”24  This  problem  is  severely  exacerbated  by  the  rapid 
rate  of  advance  in  electronics.  Technologies  are  usually  selected  and  “frozen”  early  in  full-scale 
development,  significantly  undermining  programs  based  on  fast-moving  electronic  and 
computing  technologies.  These  effects  can  be  seen  in  programs  such  as  the  F-22,  Joint  STARS, 
and  AW  ACS  Radar  System  Improvement  programs  (RSBP),  all  of  which  are  heavily  dependent 
on  computer  processing.  Current  processors  available  operate  significantly  faster.  As  a  point  of 
reference  of  today’s  technology  used  in  home  computers,  the  current  Pentium  II  processor, 
operating  at  300MHz,  can  execute  the  equivalent  of  627  million  instructions  per  second  (MIPS).25 
The  specific  program  details  are  contained  in  Appendix  2. 


24  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “A  Formula  For  Action:  A  Report  to  the 
President  on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986. 

25  Based  on  analysis  of  processor  performance  from  Intel  Corporation  processor  facts  sheets  from  their  web  pages. 
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Figure  1-10:  The  Progression  of  Commercial  Computer  Processors  vs.  Those  Fielded  in 
AWACS  RSIP,  Joint  STARS  and  F-22  Aircraft 26 


Computer  processors  are  not  the  only  area  of  technology  that  is  rapidly  changing.  Other 
areas  include  digital  signal  processors,  memory,  sensors,  communication  systems,  autonomous 
control,  and  navigation  technologies.  The  rapid  advances  within  the  commercial  electronics  area 
are  driving  many  of  these  technology  advances.  Military  aircraft,  ships,  and  space  systems  rely 
heavily  on  such  electronic  systems  to  provide  communication  and  control. 

C.4.  Slow  Response  to  New  or  Emerging  Threats 

Emerging  threats  based  on  new  technology  or  a  unique  combination  of  existing 
technologies  can  pose  a  significant  challenge  to  U.S.  forces,  and  leave  them  exposed.  Closing 
the  performance  gap  and  quickly  developing  counter-systems  is  an  important  aspect  of 
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maintaining  technologically  superiority. 

One  example  of  the  U.S.  failure  to  pursue  this  strategy  has  been  development  of  a 
counter-system  to  the  Soviet  Archer  AA-1 1  off-boresight  air-to-air  missile  that  the  soviets 
deployed  in  1985.  Off-boresight  missiles  can  attack  aircraft  at  a  wider  angle  then  standard  air-to- 
air  missiles  allowing  aircraft  equipped  with  them  to  fire  on  opponents  at  greater  angles  from  the 
nose  of  the  aircraft  and  significantly  increasing  their  chances  of  killing  their  the  opponent  before 
being  shot  down.  In  response  to  the  threat  the  Israeli’s  developed  and  fielded  a  similar  missile 
the  Python  in  1993. 

The  U.S.  system  developed  in  response  to  the  new  threat  is  the  AIM-9X,  which  is  expected  to 
reach  operational  capability  in  2002,  17  years  after  the  threat  was  identified.  Currently,  no  U.S. 
fighter  has  any  off-boresight  missile  capability. 

C.5.  Slow  Response  to  Known  Safety  Problems 

The  current  development  process  is  also  often  slow  to  respond  to  identified  safety 
requirements.  Two  high-visibility  programs  that  highlighted  this  slow  response  are  the 
integration  of  Traffic  Collision  and  Avoidance  Systems  and  Global  Positioning  System 
Receivers  on  military  aircraft.  The  time  to  field  these  systems  contributed  to  a  number  of 
avoidable  aircraft  accidents.  The  lack  of  GPS  equipment  contributed  to  the  crash  of  a  T-43 
(Boeing  737)  in  Bosnia  carrying  U.S.  business  leaders  on  a  trade  mission  and  the  crash  of  a  C- 
130  Presidential  support  plane.  The  lack  of  the  Traffic  Collision  and  Avoidance  system  on 
military  transports  contributed  to  the  collision  of  a  U.S.  Air  Force  C-141  and  German  C-130  off 
the  coast  of  Africa  in  1997. 

C.6.  Effects  of  Development  Time  on  Cost 

Long  development  times  also  impact  the  cost  of  the  systems  that  we  buy.  It  leads  to 
higher  development  cost  and  less  money  being  spent  on  producing  the  products.  Conventional 
wisdom  indicates  that  the  longer  the  development  time,  the  more  a  project  will  cost.  The 


26  Source  F-22,  AW  ACS,  and  Joint  STARS  Program  Offices. 
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Packard  Commission  concurred,  stating  that  “Time  is  money,  and  experience  argues  that  a  ten- 
year  acquisition  cycle  is  clearly  more  expensive  than  a  five-year  cycle.””  There  is  significant 
evidence  of  this  effect  in  commercial  development  efforts  “  Unfortunately,  no  data  estimating 
the  cost  of  different  development  schedules  are  available  for  specific  military  projects.  The  cost 
models  now  used  by  DoD  and  the  services  do  not  account  for  the  effects  of  time. 

Data  on  major  defense  acquisition  projects  available  from  Rand  show  a  positive 
correlation  between  development  time  and  cost.  Of  the  AC  AT  I  programs,  those  that  take  less 
than  7  years  to  complete  have  an  average  development  cost  of  $1.2  billion.  Projects  that  take 
between  7  and  14  years  have  an  average  cost  of  $1.8  billion.  Those  taking  over  14  years  average 
$3.6  billion  in  development  cost.” 


”  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “A  Formula  For  Action:  A  Report  to  the 
President  on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.p.  8. 

28  See  discussion  in  Chapter  2. 

”  For  the  123  projects  in  the  Rand  database  with  the  necessary  information,  the  Pearson  correlation  coefficient 
between  the  length  of  project  schedules  and  cost  of  development  is  positive  0.25,  with  a  two-tailed  significance 
level  of  0.005.  This  indicates  that  though  there  is  considerable  scatter  in  the  data,  longer  programs  on  average  do 
cost  more. 
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Figure  1-11:  Cost  by  Years  in  Development  for  Major  Defense  Acquisition  Programs  (Data 
from  Rand  S AR  Database) 


Similarly,  cost  and  schedule  data  on  154  projects  of  all  sizes  included  in  the  surveys 
conducted  as  part  of  this  research  effort  indicate  that  the  correlation  between  development  time 
and  cost  is  both  positive  and  statistically  significant.30 

More  Funds  Towards  Development  -  Less  Towards  Production 

Not  only  do  longer  programs  cost  more  to  develop  but  a  larger  percentage  of  the  total 
project  cost  is  consumed  during  development.  Data  from  the  Rand  S  AR  database  indicate  that 
projects  requiring  less  than  14  years  of  development  time  saw  27  percent  of  their  cost  go  to 
development  and  73  percent  go  to  production.  Project  with  development  times  over  14  years  had 
46  percent  of  their  cost  go  to  development  and  54  percent  go  to  production. 


30  Please  see  appendix  2  for  additional  details. 
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Development  Time 


Figure  1-12:  Percentage  of  Program  Cost  Going  to  Development  and  Production,  by 
Development  Time3' 


As  development  programs  have  lengthened  over  the  years,  so  has  the  percentage  of  DoD 
funding  going  to  research,  development,  test,  and  evaluation  (RDT&E).  RDT&E  funding  for 
1997  is  $32  billion,  representing  42  percent  of  total  DoD  investment  funding  (RDT&E  plus 
production).  This  is  the  highest  percentage  ever.  The  rising  percentage  spent  on  RDT&E  means 
that  a  smaller  percentage  is  available  for  producing  new  systems  or  enhancing  the  operations, 
training,  and  readiness  of  existing  forces.  While  both  RDT&E  and  production  funds  have  been 
cut  significantly  in  the  last  10  years,  the  RDT&E  accounts  have  sustained  smaller  decreases. 

This  has  resulted  in  fewer  new  systems  being  fielded  and  made  available  for  the  warfighter. 

C.7  Increased  Program  Instability  and  Cancellations 
Increased  Program  Instability 

No  only  are  long  development  times  associated  with  higher  costs  but  the  costs  are  less 
certain.  Analysis  of  the  Rand  SAR  database  indicates  that  longer  programs  typically  have  larger 


31  Source  of  Data:  Rand  SAR  Data  Base 
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percentages  of  cost  growth  then  shorter  programs.  Programs  taking  less  then  7  years  to  reach 
first  operational  delivery  overrun  their  initial  planned  development  budgets  on  average  15%. 
Programs  taking  longer  then  14  years  on  average  overrun  their  development  budgets  by  42%.32 
This  leads  to  additional  program  instability  effecting  this  and  other  programs. 

1.5 
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Average  i  « 

Cost  1 

Growth  1 .2 

Factor 

1.1 
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Figure  1-13:  Average  Program  Development  Cost  Growth  by  Development  Time. 

Analysis  based  on  data  from  the  RAND  SAR  database. 

Increased  Program  Cancellations 

Long  development  times  also  appear  to  increase  the  probability  that  a  program  will  be 
canceled  before  entering  production.  All  efforts  obtain  a  information  on  canceled  programs  from 
the  Pentagon  were  unsuccessful.  The  only  data  found  comes  from  the  book  Augustine’s  Laws. 
According  to  Norm  Augustine’s, analysis  of  1 14  canceled  programs,  each  program  stands  about  a 
4.4%  chance  per  year  of  being  canceled.  Efforts  to  obtain  the  names  and  data  on  the  canceled 
programs  were  unsuccessful.  However,  in  a  later  article  he  cites  representative  examples  of 
canceled  programs  including  four  canceled  Army  air  defense  systems:  Mauler,  Roland,  Sgt  York, 
and  ADATS.  None  of  which  were  fielded.  These  systems  cost  $6.7  billion  in  development  costs 
and  produced  no  combat  capability.33 


<  7  Years  7-1 3  Years  >1 4  Years 

Time  to  First  Operational  Delivc 


32  Jarvaise,  Drezner,  and  Norton.  “The  Defense  System  Cost  Performance  Database:  Cost  Growth  Analysis  Using 
Selected  Acquisition  Reports.”  MR-625-OSD  Santa  Monica  CA:  The  RAND  Corporation.  1996. 

33  Augustine,  Norman.  “From  Industry... Acquisition  Reform:  Dream  or  Mirage.”  Army  RD&A  Magazine. 
September-October  1996.  Pg.  20-22. 
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Figure  1-14:  Cumulative  Probability  of  Cancellation  of  Defense  Programs  Over  Time.34 

Norm  Augustine  indicates  the  cost  of  these  canceled  programs  in  terms  of  lost  military 
capability.  He  states  the  funds  expended  on  the  canceled  “could  have  purchased  1,000  Abrams 
tanks,  100  F-16  Fighters,  1,000  AMRAAM  Missiles,  10  Titan  IV  Rockets,  20  JSTARS,  10,000 
Javelin  Missiles,  70,000  MLRS  Rockets,  and  One  Nuclear  Attack  Submarine”35 

Increased  Management  Turnover 

One  potential  reason  for  the  increased  program  instability  and  increased  cancellations  is 
the  difficulty  maintaining  a  consensus  among  a  large  number  of  program  and  service  leaders  over 
time.  As  shown  in  the  table  below,  the  average  program  taking  1 1  years  to  go  from  program 
initiation  to  initial  operational  capability  has  experienced  a  significant  amount  of  management 
turn  over. 


54  Augustine,  Norman  R.  Augustine’s  Laws.  American  Institute  of  Aeronautics  and  Astronautics.  Washington  DC. 
1983. 

35  Augustine,  Norman.  “From  Industry... Acquisition  Reform:  Dream  or  Mirage.”  Army  RD&A  Magazine. 
September-October  1996.  Pg.  20-22. 
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Number  of:  (132  Months  Avg  ACAT I)* 


Program  Director  4 

Program  Executive  Officer  5 

Service  Acquisition  Executive  8 

Defense  Acquisition  Executive  8 

Chairman  of  the  Joint  Chiefs  of  Staff  5 

Secretary  of  Defense  7 

Presidents  3 

Budget  Cycles  11 


Table  1-3:  Management  Turnover  for  the  Average  Development  Time  for  a  Major  Defense 
Acquisition  Programs.* 


C.8  Increased  Sustainment  Costs 

Long  development  times  also  contribute  to  increased  cost  of  operations  and  sustainment. 
It  increases  the  time  required  to  replace  systems  with  high  operating  costs.  One  example  is  the 
DD-21  destroyer  that  is  expected  to  lower  the  operating  cost  of  a  Navy  destroyer  by  70% 
compared  to  today’s  destroyers.  Long  development  times  increase  the  time  to  replace  hard  to 
maintain  systems  such  as  the  current  F-15  Radar  which  has  a  12  hours  Mean  time  between 
failure.  The  upgraded  radar  systems  under  development  are  expected  to  have  120  hours  mean 
time  between  failure.37 

Long  development  times  also  increased  the  impact  of  diminishing  manufacturing  base 
parts  and  with  long  development  times  the  problems  occur  earlier  in  a  systems  life.  One  example 
is  F-22  which  is  approaching  its  production  decision  currently  has  593  parts  which  are  already 
out  of  production.  Replacing  these  parts  in  the  design  is  expected  to  cost  $279M.38 

Another  way  that  long  development  times  impact  sustainment  of  systems  is  that  once  a 
new  program  is  started  it  typically  freezes  upgrades  to  the  existing  systems.  This  is  done  in  part 
to  ensure  a  significant  difference  between  the  new  system  and  the  old  system  to  justify  the  new 


36  Developed  by  looking  at  management  turn  over  in  key  offices  for  the  last  1 1  years. 

37  Information  obtained  from  the  F-15  Program  Office  in  March  98. 

38  Information  obtained  from  the  F-22  Program  Office  in  March  1998. 
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system.  Upgrades  and  modifications  to  existing  systems  must  compete  for  the  same  scare 
resources  with  new  programs.  Many  examples  exist  including  fighter  Aircraft,  satellite 
communications,  bombs  and  missiles.  Prior  to  the  delivery  of  the  new  systems  or  in  the  event 
the  new  system  is  canceled  the  warfighter  is  left  with  less  then  optimal  equipment. 

% 

C.9  Other  Impacts  of  Long  Development  Time 

There  are  many  other  potential  impacts  of  long  development  time.  A  sample  will  be 
mentioned  but  not  covered  in  detail.  The  analysis  is  contained  in  the  appendix  or  left  for 
subsequent  work. 

Increase  requirements  and  specification  due  to  uncertain  threat  forecasts 

Increased  technology  step  sizes 

Increased  number  of  programs  in  development  at  once 

Increased  competition  for  resources 

Decreased  management  attention  per  project 


D.  Efforts  to  Shorten  Development  Times 

While  there  have  been  a  flurry  of  acquisition  reform  activity  in  recent  years,  little  of  this 
effort  has  been  aimed  primarily  at  reducing  development  time.  The  primary  aim  of  the  current 
acquisition  reform  efforts  has  been  focused  on  lowering  costs.  Acquisition  reform  initiatives  that 
focus  primarily  on  cost  include  cost  as  an  independent  variable  (CATV),  elimination  of  military 
specification  and  standards,  single  process  initiative  (SPI),  performance  based  specifications, 
clear  accountability  in  design,  and  the  manufacturing  development  initiative.  These  initiatives 
may  also  effect  development  time,  though  indirectly.  A  few  efforts  have  aimed  at  shortening 
acquisition  schedules  including  the  Packard  Commission  and  the  Affordable  Acquisition 
Approach  Study.  These  efforts  are  outlined  below. 
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D.l.  Previous  Efforts 

Many  reports,  teams,  and  commissions  have,  in  one  way  or  another,  attempted  to  address 
the  DoD  development  problem.  Most  have  focused  on  long  production  schedules  and  high  costs. 
Many  have  focused  on  the  inability  to  meet  projected  schedules.  Those  efforts  include  the 
influential  Packard  Commission  and  the  current  National  Performance  Review.  But  although 
reform  efforts  have  focused  on  long  development  times  since  the  late  1970s,  none  appear  to  have 
had  significant  effect. 

Affordable  Acquisition  Approach  Study 

The  Air  Force  Systems  Command  Affordable  Acquisition  Approach  completed  in  1983 
focused  on  two  questions:  are  projects  taking  longer  and  what  can  be  done  about  it.  The  study 
found  that  development  times  had  increased  significantly  over  the  previous  30  years.  The  study 
also  found  that  the  major  cause  of  lengthening  development  and  production  times  was  the  over 
commitment  of  resources  within  the  Air  Force  budget.39  The  emphasis  on  development  time 
decreased  as  the  study  progressed  and  is  evident  in  the  change  of  the  projects  name  from  its 
original  name  as  the  Accelerated  Acquisition  Approach  Study  to  the  Affordable  Acquisition 
Approach  Study.  One  notable  participant  was  the  lead  contract  leader  was  Dr  Jacque  Gansler,  the 
current  Defense  Acquisition  Executive.  Few  identifiable  actions  resulted  from  the  study. 


The  Packard  Commission 

The  1986  Packard  Commission  looked  at  the  entire  defense  acquisition  process,  citing 
long  development  times  as  the  central  problem  from  which  most  other  acquisition  problems 
originate.  The  Packard  Commission’s  stark  assessment  identified  problems  at  all  levels,  from 
program  managers  to  Congress.  Its  major  recommendations  were  to  cut  acquisition  time  in  half 
by  emulating  successful  commercial  firms  with  world-class  customers.  In  its  Formula  for 
Action,  the  Packard  Commission  stated: 

Acquisition  problems  have  been  with  us  for  several  decades,  and  are  becoming 
more  intractable  with  the  growing  adversarial  relationship  between  government  and  the 
defense  industry,  and  the  increasing  tendency  of  Congress  to  legislate  management 


39  Air  Force  Systems  Command.  “Affordable  Acquisition  Approach”.  Andrews  AFB  MD.  1983 
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solutions.  In  frustration,  many  have  come  to  accept  the  ten-to-fifteen  years  acquisition 
cycle  as  normal,  or  even  inevitable. 

We  believe  that  it  is  possible  to  cut  this  cycle  in  half.  This  will  require  radical 
reform  of  the  acquisition  organization  and  procedures.  It  will  require  concerted  action  by 
the  Executive  Branch  and  Congress,  and  full  support  of  the  defense  industry. 

Specifically,  we  recommend  that  the  administration  and  the  Congress  join  forces  to 
implement  the  following  changes  in  the  defense  acquisition  system. 

Packard  Commission  Formula  for  Action  198640 

To  achieve  this  goal,  the  commission  recommended  streamlining  acquisition 
organizations  and  procedures,  using  technology  to  reduce  costs,  balancing  performance  with 
costs,  stabilizing  programs,  expanding  the  use  of  commercial  products,  increasing  competition, 
and  raising  the  quality  of  acquisitions  personnel.  The  commission  further  aimed  to  consolidate 
acquisition  efforts  under  the  Office  of  the  Secretary  of  Defense  (OSD),  create  acquisition 
executives  at  the  assistant  secretary  level  for  each  service,  and  rely  on  executive  officers  to 
oversee  major  projects  within  the  services.  The  commission  also  established  rigid  objectives  for 
cost,  schedule,  and  performance  for  all  major  programs,  and  advocated  standards  and  training 
programs.  President  Reagan  quickly  accepted  the  commission’s  recommendations,  and 
Congress,  which  had  been  eager  for  acquisition  reform,  quickly  turned  them  into  law. 

With  the  bold  charge  to  cut  acquisition  time  in  half,  strong  support  from  the  president  and 
Congress,  and  quick  response  by  DoD,  one  would  have  expected  dramatic  results.  However, 
although  DoD  implemented  the  commission’s  recommendations,  it  did  not  widely  internalize  the 
goal  of  slashing  development  time.  Few  of  the  people  I  interviewed  realized  that  reducing 
development  time  was  even  a  significant  objective  of  the  Packard  Commission.  The  current 
focus  of  acquisition  reform  efforts  is  clearly  on  cutting  costs,  not  reducing  development  time. 
Only  a  few  of  the  hundred  or  so  current  reform  initiatives  are  aimed  at  reducing  development 
schedules. 


40  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “A  Formula  For  Action:  A  Report  to  the 
President  on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.  Pg.  15. 
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D.2  Recent  Efforts  to  Shorten  Product  Cycle  Time 

Nevertheless,  three  recent  efforts  do  directly  address  product  development  time:  Air 
Force  Acquisition  Reform  Initiative  (“Lightening  Bolt  #10”),  the  DoD  Advanced  Concept 
Technology  Demonstrations  (ACTDs),  and  the  Lean  Aerospace  Initiative,  a  consortium 
involving  industry,  government,  and  academia  of  which  this  research  is  a  part.  Two  new 
initiatives,  the  Defense  Systems  Affordability  Council  Acquisition  Cycle  Time  Reduction  Tack 
Force,  and  the  Air  Force  Cycle  Time  Reduction  Tiger  Team,  are  focused  directly  on  reducing 
development  time. 

AF  Lightning  Bolt  Initiative  #10 

Just  after  his  confirmation  in  1996  as  assistant  secretary  for  acquisition,  Arthur  Money 
began  a  new  initiative  to  cut  the  time  to  develop  and  field  new  Air  Force  systems  time  in  half. 
The  description  of  Lightning  Bolt  #10  in  March  1996  read: 

Lightning  Bolt  #10.  The  time  from  initial  effort  by  a  buying  office  to  satisfy  a  user's 
validated  requirements  (for  a  new  product,  services,  parts,  etc.)  until  delivery  will  be 
reduced  by  50% 41 

AFAR  April/March  1996 

However,  the  project’s  scope  was  soon  narrowed  to  cutting  the  time  from  receipt  of  requirements 
and  allocated  funds  to  contract  award  in  half.  The  acquisitions  community  therefore  limited  the 
complicating  factors  and  focused  only  on  those  parts  under  its  control.42  The  objective  of  the 
initiative  was  changed  to  read: 

Reduce  by  50%  the  amount  of  time  to  award  contracts  that  meet  our  customers’  needs. 
This  time  begins  with  receipt  of  a  validated  user  requirement  and  funding  commitment, 
and  ends  with  contract  award.  Lightning  Bolt  #10  applies  to  efforts  to  develop  and 
acquire  systems,  and  support  their  operational  readiness.43 

AFAR  June/July  1996 


41  Air  Force  Acquisition  Reform  Online  newsletter  March/April  1996. 

42  Personal  discussion  with  Col  Ben  McCarter.  Lightening  Bolt  10  Leader.  13  June  1996. 

43  Air  Force  Acquisition  Reform  Online  newsletter  June/July  1996. 
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The  Lightning  Bolt  #10  group  conducted  interviews  with  program  managers  and 
documented  a  set  of  best  practices  and  new  ideas  to  reduce  time  to  contract  award.  The 
Lightning  Bolt  #10  group  disbanded  after  issuing  its  report  and  placing  its  “tool  box”  of  ideas  on 
the  Internet. 

Advanced  Concept  Technology  Demonstrations  (ACTDs) 

Advanced  Concept  Technology  Demonstrations,  an  initiative  of  the  Office  of  the 
Secretary  of  Defense  (OSD),  are  designed  to  reveal  the  utility  of  readily  available  technologies 
for  meeting  pressing  military  needs.  The  idea  is  to  allow  the  warfighting  community  to  evaluate 
a  technology’s  military  utility  before  committing  to  a  major  development  effort..  The  OSD 
expects  these  demonstration  programs  to  last  between  2  and  4  years. 

Paul  Kaminski,  former  undersecretary  of  defense  for  acquisition  and  technology, 
envisioned  the  program  as  a  way  to  compress  the  time  required  to  develop  and  field  weapon 
systems  and  to  stimulate  innovations  needed  to  implement  a  revolution  in  military  affairs.'14 

ACTDs  are  expected  to  reduce  cycle  time  by  allowing  an  acquisition  process  to  begin  at 
Milestone  II — the  beginning  of  full-scale  development.  Former  Undersecretary  Kaminski  also 
points  out  that  because  they  are  not  part  of  the  official  acquisition  process,  ACTD  projects  can 
incorporate  considerably  more  flexibility  in  their  contracts  as  they  do  not  fall  within  the  formal 
rules  of  an  acquisition  program  and  are  often  able  to  use  a  different  procurement  category.  After 
completing  an  ACTD,  the  warfighting  commanders  can  recommend  proceeding  to  low-rate 
initial  production,  pursue  additional  demonstration  to  improve  the  technology’s  performance,  or 
drop  the  technology. 

DoD  now  provides  a  financial  incentive  for  services  to  conduct  ACTDs  by  adding  10 
percent  to  what  a  service  commits,  although  the  number  of  ACTDs  is  limited  by  total  DoD 
funding.  As  of  1997,  some  15  small  development  efforts  are  operating  under  the  ACTD  model— 


44  Kaminski,  Dr.  Paul  G.  “Advanced  Concept  Technology  Demonstrations:  Challenges  and  Opportunities’ 
Keynote  Address,  ACTD  Manager’s  Conference,  DSMC,  Fort  Belvoir,  VA,  Sept  10,  1996. 
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a  small  but  visible  effort.  However,  this  approach  is  intended  to  circumvent  the  official 
development  process,  not  change  the  process  used  to  develop  most  systems  and  projects. 


The  Lean  Aerospace  Initiative  (LAI) 

The  Lean  Aerospace  Initiative,  a  consortium  encompassing  industry,  government,  labor, 
and  academia,  is  led  by  the  Massachusetts  Institute  of  Technology,  where  17  professors,  9  full¬ 
time  research  staff,  and  over  20  graduate  student  research  assistants  work  under  the  LAI  purview. 
The  program  is  intended  to  reduce  cost,  development,  and  production  time  for  military  products 
by  half  by  infusing  commercial  lean  practices  throughout  the  defense  aerospace  industry. 
Participants  are  conducting  research  in  all  phases  of  the  development  and  manufacturing  process, 
including  =  factory  operations,  supplier  relations,  and  government  policy. 

This  approach  contrasts  with  most  efforts  to  reduce  development  time,  which  often  focus 
only  on  certain  aspects.  The  figure  below  maps  the  cycle  time-related  initiatives  against  the 
phases  of  the  development  effort  that  they  effect. 
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Hm  Required 

Idea  or  Need  Decision  to  Production  Asset 


Figure  1-15:  Mapping  of  Current  Cycle  Time  Reduction  Efforts 


D.3  New  Cycle  Time  Initiatives 

In  the  late  fall  of  1997,  two  new  programs,  one  at  the  DoD  level  and  one  at  the  Air  Force 
level,  were  initiated  to  help  address  long  development  time.  DoD’s  Acquisition  Cycle  Time 
Reduction  Task  Force  aims  to  cut  the  period  from  project  initiation  to  initial  operational 
capability  for  major  systems  in  half.  The  Air  Force  Cycle  Time  Reduction  Team  aims  to 
dramatically  cut  the  time  to  develop  and  field  new  and  modified  systems.  This  research  has 
shaped  a  significant  portion  of  these  efforts.45 


45  Note:  The  author  is  the  Air  Force  Representative  on  the  DoD  Cycle  Time  Reduction  Task  Force  and  the  Co¬ 
leader  of  the  Air  Force  Cycle  Time  Reduction  Team. 
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Chapter  Summary 

The  Packard  Commission  identified  long  cycle  time  as  the  central  problem  in  the 
acquisition  process.  Development  times  for  military  systems  have  increased  significantly  over 
the  last  30  years.  Current  development  times  for  major  defense  acquisition  program  is 
approaching  1 15  months.  Time  from  program  initiation  to  initial  operating  capability  (when  the 
system  is  ready  for  use)  is  even  longer.  The  time  required  to  develop  or  modify  major  military 
systems  is  now  approaching  1 1  years.  Long  development  cycles  affect  warfighters  as  systems 
long  in  development  are  not  available  for  use,  the  technology  included  in  these  systems  when 
fielded  is  dated,  the  time  to  meet  new  threats  is  long,  and  the  time  to  fix  identified  problems  is 
long.  Long  development  times  also  impact  the  cost  of  developing  and  maintaining  weapon 
systems.  Long  development  times  also  cause  program  uncertainty  as  costs  are  less  certain,  more 
programs  are  canceled,  and  leadership  changes  as  do  leadership  priorities. 

While  several  studies  have  attempted  to  identify  the  causes  of  long  cycle  times,  they  have 
had  little  effect,  as  development  times  have  continued  to  grow.  Part  of  the  problem  is  that  most 
of  the  reform  initiatives  focus  solely  on  cutting  the  cost  of  weapons.  Few  of  the  current 
initiatives  focus  on  reducing  development  or  acquisition  response  times.  What’s  more,  two  of 
the  three  initiatives  to  reduce  development  time  address  only  part  of  the  problem.  Clearly  much 
needs  to  be  done  if  the  time  required  to  develop  new  military  systems  is  to  be  significantly 
reduced. 


Chapter  2 


Commercial  Efforts  to  Reduce  Product 
Development  Time _ 

A.  “An  Acquisition  Model  to  Emulate” 

Commercial  companies  and  practices  are  often  held  up  as  a  model  for  the  DoD  acquisition 
system.  The  Packard  Commission,  for  example,  referred  specifically  to  an  analysis  by  the  Defense 
Science  Board  of  large  commercial  development  programs.  The  board  evaluated  several  multi¬ 
year,  multi-billion-dollar  programs  comparable  to  the  complex  efforts  required  to  develop  major 
military  systems.  The  commercial  projects  included  the  IBM  360  personal  computer,  the  Boeing 
767  transport,  the  AT&T  telephone  switch,  and  the  Hughes  communications  satellite.  The  Defense 
Science  Board  study  found  that  these  projects  were  completed  in  half  the  time  required  for  similar 
DoD  development  efforts,  and  cost  concomitantly  less.*  The  Packard  Commission  noted  that 
“these  commercial  programs  clearly  represent  the  models  of  excellence  we  are  seeking . . .”  The 
commission  also  cited  several  defense  projects,  such  as  the  Polaris  and  Minuteman  missiles  that 
had  achieved  the  accelerated  schedules  of  commercial  programs.  Based  on  these  results,  the 


46  President’s  Blue  Ribbon  Commission  on  Defense  Management.  A  Quest  For  Excellence:  A  Report  to  the 
President  on  Defense  Acquisition  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.  Page  11. 
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commission  concluded  that  it  was  possible  to  apply  commercial  lessons  to  the  Department  of 
Defense.  The  commission  stated: 

It  is  clear  that  major  saving  are  possible  in  the  development  of  weapons  systems  if  DoD 
broadly  emulates  the  acquisition  procedures  used  in  outstanding  commercial  programs.  In 
a  few  programs,  DoD  has  demonstrated  that  this  can  be  done.  The  challenge  is  to  extend 
the  correct  management  techniques  to  all  major  defense  acquisitions,  and  more  widely 
realize  the  attendant  benefits  in  schedule  and  costs.47 


The  commission  identified  six  management  features  necessary  to  cut  cycle  time  and  cost:  clear 
command  channels,  program  stability,  limited  reporting  requirements,  small  but  high-quality 
staffs,  good  communication  with  end  users,  and  effective  prototyping  and  testing.  The 
Commission  made  specific  recommendations  in  these  areas,  which  were  implemented.  But,  as 
we  have  seen,  development  times  for  military  systems  have  not  decreased,  indicating  either  a 
problems  with  implementing  the  recommendations  or  other  critical  factors  associated  with 
commercial  projects  not  addressed  by  the  Packard  Commission.  Dramatic  changes  have 
occurred  in  the  commercial  product  development  world  since  the  Defense  Science  Board  and 
Packard  Commission  issued  their  reports.  More  efficient  and  effective  development  time  has 
become  the  competitive  focus  of  many  of  the  most  successful  firms. 

B.  Competing  on  Product  Development  Time 

In  the  last  15  years,  the  time  required  to  develop  and  market  commercial  products  has 
been  dramatically  reduced.  Firms  are  competing  not  only  on  price  but  also  on  their  ability  to 
quickly  produce  high-quality  products  that  meet  the  changing  needs  of  their  customers. 

Commercial  firms  have  found  that  by  reducing  time  to  market,  they  can  also  lower  costs. 
This  has  allowed  companies  to  expand  the  number  of  new  products  they  develop.  The  result  is  a 
wider  array  of  products  that  cost  less  than  their  predecessors. 

Companies  with  fast  product  development  abilities  have  a  number  of  options  open  to 
them.  They  can  deliver  a  product  to  market  before  their  competitors,  thereby  capturing  market 


47  President’s  Blue  Ribbon  Commission  on  Defense  Management.  A  Quest  For  Excellence:  A  Report  to  the 
President  on  Defense  Acquisition  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.  Page  12. 
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share  and  setting  industry  standards.  Alternatively,  they  can  choose  to  start  a  project  after  a 
competitor  but  deliver  it  to  market  at  the  same  time-with  more  cutting-edge  technology  or 
specific  attributes  that  meet  customers’  needs.  Fast  product  development  times  also  allow  a 
company  to  quickly  respond  to  a  new  product  introduced  by  a  competitor.  The  ability  to  develop 
and  manufacture  products  quickly  allows  a  company  to  select  from  a  number  of  competitive 
strategies  not  available  to  companies  with  significantly  longer  development  times. 

A  number  of  books  have  recently  appeared  on  the  subject  of  reducing  product 
development  time.  Two  of  the  more  popular  are  Revolutionizing  Product  Development,  by  Kim 
Clark,  and  New  Product  and  Process  Development ,  by  Steven  Wheelwright.  Others  include 
Developing  Products  in  Half  the  Time,  Lightning  Strategies  for  Innovation,  Competitiveness 
Through  Total  Cycle  Time,  Fast  Cycle  Time,  and  Survival  of  the  Fittest:  New  Product 
Development  for  the  90s.41  All  these  books  focus  on  reducing  cycle  time  as  a  critical  method  for 
improving  commercial  performance. 

Competing  on  time  to  market  is  not  limited  only  to  consumer  products  some  companies 
in  the  defense  industry  also  rely  on  this  approach.  Firms  involved  in  communications,  satellites, 
computers,  and  aircraft,  for  example,  have  all  cut  product  development  times. 

Of  all  these,  the  automobile  industry’s  efforts  to  reduce  product  development  times  are 
the  most  thoroughly  documented.  Major  research  on  the  automobile  industry,  such  as  the  MIT 
International  Motor  Vehicle  Program  and  the  Harvard  Automobile  Study,  has  revealed  detailed 
information  on  how  these  efforts  have  succeeded. 


C.  Reducing  Product  Development  Time  in  the  Automobile  Industry 

In  the  late  1980s,  Japanese  automobile  companies  maintained  a  substantial  lead  over 
American  and  European  companies  in  product  development  and  manufacturing.  The  Japanese 
were  able  to  develop  and  produce  higher-  quality  cars,  with  more  new  parts,  in  significantly  less 
time,  with  significantly  fewer  people,  at  significantly  lower  cost,  than  their  U.S.  competitors. 
These  advantages  allowed  the  Japanese  to  offer  significantly  more  new  models  and  model 


48  For  full  bibliography  information,  please  see  the  reference  section  in  Appendix  5. 
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upgrades  every  year,  and  to  rapidly  include  new  features  and  technologies  demanded  by 
customers.  This,  in  turn,  enabled  them  to  dramatically  increase  their  market  share  at  the  expense 
of  their  U.S.  competitors,  who  struggled  for  their  very  survival.  Table  2-1  provides  a  view  of  the 
product  development  performance  of  Japanese  and  American  automobile  manufacturers  in  the 
mid-1980s.  The  table  shows  that  the  Japanese  developed  new  models  using  only  68  percent  of 
the  time  required  by  U.S.  companies.  This  allowed  the  Japanese  to  start  a  new  development 
project  one  and  a  half  years  later  than  an  average  American  company  yet  bring  it  to  market  at  the 
same  time. 

Japanese  companies  developed  equivalent,  if  not  superior,  cars  with  roughly  one-third  the 
engineering  hours  and  used  only  485  engineers  per  project  compared  with  903  at  U.S.  firms-a 
figures  that  account  for  the  majority  of  development  costs.  Ratings  by  services  such  as  JD 
Powers  and  Associates  attested  to  the  significantly  higher  quality  of  the  Japanese  vehicles. 
Japanese  companies  also  found  not  only  that  they  could  undertake  more  development  efforts 
with  a  given  level  of  resources  but  that  they  maintained  more  control  over  their  development 
efforts.  The  result  was  significantly  fewer  delayed  products  (one  in  six)  than  their  American 
counterparts  (one  in  two). 
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Automobile  Producers 

.Tanan 

U.S. 

Average  Development  Time  (Months) 

42.6 

62.0 

Average  Engineering  Time  (Millions  of  Hours) 

1.2 

3.5 

Total  Product  Quality  (Rating) 

58 

41 

Number  of  Employees  in  Project  Team 

485 

903 

Number  of  Body  Types  per  New  Car 

2.3 

1.7 

Average  Ratio  of  Shared  Parts 

18% 

38% 

Supplier  Share  Engineering 

51% 

14% 

Engineering  Change  Costs 

10-20% 

30-50% 

as  share  of  total  die  cost 

Ratio  of  Delayed  Products 

1  in  6 

1  in  2 

Return  to  normal  quality  after  new  model  (months) 

1.4 

11 

Table  2-1:  Product  Development  Performance  of  U.S.  and  Japanese  Auto  Industries  (Mid- 

1980s)49 


When  analyzing  the  performance  of  Japanese  and  American  companies,  for  example, 

Kim  Clark  and  Takahiro  Fujimoto50  found  that  the  Japanese  were  not  “buying”  lead  time  by  using 
additional  resources,  as  had  been  expected.51  Instead,  they  found  a  positive  correlation  between 
speed  and  efficiency:  faster  firms  were  more  efficient  and  slower  firms  were  less  efficient.52  This 
result  may  have  been  partly  due  to  the  ability  to  use  the  design-build-test  cycle  to  quickly 
identify  and  eliminate  problems.  Clark  and  Fujimoto  noted: 

Time-To-Market  is  such  a  critical  dimension  of  performance  in  the  outstanding  project, 
that  all  of  the  processes,  systems,  and  activities  are  geared  to  fast  action.  This  is 
particularly  true  for  the  critical  design-build-test  cycles  that  are  at  the  heart  of  problem 
solving  in  development . . .“ 


49  Composite  data  from  Product  Development  Performance  and  Womack,  Jones,  and  Roos.  The  Machine  that 
Changed  the  World:  The  Story  of  Lean  Production.  Nev  York:  Harper  Perennial.  1990.  Pg.  118. 

50  Clark,  Kim  and  Takahiro  Fujimoto.  Product  Development  Performance:  Strategy.  Organization  and  Management 
in  the  World  Automobile  Industry.  Cambridge  MA:  Harvard  University  Press.  1991. 

51  The  idea  of  buying  lead-time  was  wide  spread  in  DoD  Programs  and  is  called  “crashing  a  program”  by  applying 
additional  resources  to  shorten  the  schedule. 

52  Clark,  Kim  and  Takahiro  Fujimoto.  Product  Development  Performance:  Strategy.  Organization  and  Management 
in  the  World  Automobile  Industry.  Cambridge  MA:  Harvard  University  Press.  1991.  Pg.  87. 

55  Clark,  Kim  and  Takahiro  Fujimoto.  Product  Development  Performance:  Strategy.  Organization  and  Management 
in  the  World  Automobile  Industry.  Cambridge  MA:  Harvard  University  Press.  1991.  Pg.  304. 
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Japanese  automakers  used  their  product  development  ability  to  increase  the  number  of 
models  they  offered  from  47  in  1982  to  84  in  1990.  American  automobile  manufacturers,  in 
contrast,  increased  their  offerings  from  36  in  1982  to  53  models  in  1990.  During  this  same 
period,  the  Japanese  were  replacing  their  models  on  average  every  4  years,  while  the  American 
manufacturers  were  replacing  their  models  on  average  every  10  years. 

By  updating  their  models  more  quickly,  the  Japanese  were  able  to  adapt  and  market  new 
technologies  faster  than  U.S.  companies.  They  also  could  quickly  incorporate  new  and  cheaper 
manufacturing  processes,  and  build  on  lessons  from  the  previous  generation.  Japanese  product 
development  performance  allowed  them  to  enter  market  areas  in  force,  such  as  the  luxury  car 
market,  with  new  lines  such  as  the  Lexus  and  Infinity,  that  compete  effectively  with  established 
brands  in  this  profitable  sector. 

The  Japanese  focus  on  reducing  development  time  played  a  central  role  in  this  success. 
Clark  and  Fujimoto  indicate  the  impact  of  a  focused  effort  on  cutting  development  cycle  time: 

Faster  Development  Time  -  A  Unifying  Driver.  Just  as  engineers  need  a  vision  of  the 
overall  product  to  guide  their  efforts  in  developing  a  new  car,  the  people  involved  in 
changing  the  development  organization  need  a  vision,  an  objective  that  captures  their 
imagination.  Where  changes  have  taken  hold  and  worked,  senior  managers  have  linked 
the  need  for  a  new  organization  to  competition  and  the  drive  for  tangible  results  in  the 
market  place.  The  quest  for  faster  development  lead-time  has  been  a  particularly 
powerful  driver  of  this  effort  during  the  1980s.  Lead-time  is  not  an  end  in  itself,  but  its 
pursuit  leads  people  to  do  things  that  improve  the  system  overall.  In  respect,  lead  time  is 
like  inventory  in  a  Just-In-Time  manufacturing  system;  of  itself,  a  low  level  of  work  in 
progress  inventory  has  some  effect,  but  going  after  the  root  causes  of  excess  inventory 
brings  about  powerful  system  changes.”54 


Finding  themselves  at  a  severe  competitive  disadvantage,  U.S.  automakers  responded 
quickly  to  the  crisis  in  the  early  1990s  and  significantly  reduced  their  product  development 
times,  which  are  now  are  now  less  than  two  years  for  a  new  car.  The  variety  of  U.S.  vehicles  has 
dramatically  increased,  and  quality  has  never  been  higher.  Cars  are  lasting  longer,  and  their 
costs  are  actually  declining. 


54  Clark,  Kim  and  Takahiro  Fujimoto.  Product  Development  Performance:  Strategy.  Organization  and  Management 
in  the  World  Automobile  Industry.  Cambridge  MA:  Harvard  University  Press.  1991.  Pg.  282 
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D.  Reducing  Product  Development  Time  in  Other  Industries 

Automobiles  are  not  the  only  industry  to  significantly  reduce  its  product  development 
time;  that  focus  is  spreading  to  nearly  every  sector.  The  driving  forces  behind  this  phenomenon 
are  intense  international  competition,  fragmented  and  demanding  markets,  and  diverse  and 
rapidly  changing  technologies.55  The  result  of  this  competition  has  been  an  explosion  in  product 
variety,  dramatic  increases  in  quality,  and  large  decreases  in  costs  over  a  wide  range  of 
industries,  from  textiles  to  consumer  electronics.  These  developments  have  also  affected 
commercial  sectors  related  to  the  defense  industry,  such  as  the  commercial  aircraft,  satellite, 
computer,  and  communications  industries. 

The  commercial  aircraft  industry  is  seeing  significant  competition  in  several  categories, 
from  business  jets,  to  regional  jets,  to  wide  body  jets,  in  part  because  of  decreases  in  product 
development  times.  Boeing  produced  the  777  in  five  years  during  the  1990s.  Boeing  has  now 
set  an  aggressive  goal  to  reduce  development  time  for  future  aircraft  to  two  and  a  half  years  for 
new  aircraft  and  18  months  for  modifications  to  existing  aircraft.56  Such  a  focus  gives  all  groups 
within  the  firm  a  clear  and  effective  measure  of  performance. 

The  commercial  communication  satellite  industry  has  also  seen  a  large  reduction  in  cycle 
times.  Hughes  Aircraft  Co.  states  that  it  can  develop  and  launch  a  new  satellite  that  responds  to  a 
specific  customer's  needs  in  as  little  as  18  months.  Previous  development  times  exceeded  five 
years.57  Similarly,  computer  companies  have  cut  product  development  cycles  to  less  than  six 
months  to  keep  up  with  rapidly  changing  technology. 


55  Clark,  Kim  and  Wheelwright,  Steven.  Revolutionizing  Development:  Quantum  Leaps  in  Speed.  Efficiency,  and 
Quality.  New  York:  The  Free  Press.  1992. 

56  Walt  Gillette,  Boeing  Commercial  Aircraft  Company  MIT  Seminar  Fall  1997. 

57  Hughes  Spacecraft.  Effort  known  as  “Project  18,”  representing  18  months  from  contract  to  on  orbit  operations. 


66  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Industry 

Old  Time 

Current 

Goal 

Automobile 

7  years 

2  years 

<18  months 

New  Commercial  Aircraft 

8-10  years 

5  years 

2 1/2  years 

Commercial  Spacecraft 

8  years 

18  months 

12  months 

Consumer  Electronics 

2  years 

6  months 

Table  2-2:  Various  Industry  Product  Development  Times  and  Goals 

Of  course,  the  purposes  of  military  systems  and  commercial  systems  differ.  The 
motivations  of  commercial  firms—profits  and  market  share-differ  from  that  of  the  DoD  of 
providing  the  most  effective  defense.  However,  the  means  to  achieve  those  objectives  are 
similar:  to  develop  and  field  the  highest-quality,  lowest-cost  systems  that  meet  customers’  needs. 
The  fast  commercial  time-to-market  competitor  can  charge  a  price  premium  for  a  new  product 
with  higher  quality  or  additional  features  until  a  competitor  matches  the  offer.  Alternatively, 
manufacturers  can  decide  to  increase  market  share  by  offering  better  performance  for  the  same 
price.  The  slow  competitor  does  not  have  these  options.  The  same  is  true  for  DoD:  if  it  can 
deliver  a  better  system  with  more  advanced  technology  that  meets  warfighters’  needs  faster  than 
an  opponent  can  field  a  counter  system,  the  U.S.  can  achieve  a  significant  military  advantage. 
This  fact  was  not  lost  on  Undersecretary  Kaminski,  who  told  a  Senate  subcommittee:  “The 
military  advantage  goes  to  the  nation  who  has  the  best  cycle  time  to  capture  technologies  that  are 
commercially  available;  incorporate  them  in  weapon  systems;  and  get  them  fielded  first.”5" 

This  view  is  shared  by  at  least  some  defense  industry  leaders  such  as  Dean  Clubb,  head  of 
Texas  Instruments’  Defense  and  Electronics  Group,  who  wrote:  “As  the  pace  of  the  world 
quickens,  the  value  of  being  first  to  market  with  innovative  solutions  is  the  key  to  true 
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competitive  advantage.  This  is  true  in  the  commercial  market  place  and  it  is  also  true  in  the 
military  market.”59 


Chapter  Summary 

The  commercial  industry  has  been  often  cited  as  an  example  for  DoD  to  emulate.  The 
commercial  industry  has  made  great  strides  in  reducing  their  development  times.  The  have  done 
this  for  competitive  reasons  and  the  requirement  to  quickly  meet  their  customers  changing  needs 
with  high  quality  and  low  cost  products.  The  focus  on  the  reduction  on  development  time  is  seen 
as  an  organizing  focus  from  which  to  organize  development  efforts.  They  have  found  that  by 
focusing  on  the  reduction  of  development  time,  they  force  dramatic  improvements  in  their 
business  processes.  This  has  resulted  in  higher  quality  products,  at  lower  cost,  in  dramatically 
less  time.  Reductions  in  development  times  are  often  between  50%  and  75%  in  many  industries, 
even  in  industries  closely  related  to  the  defense  industry. 


58  “Statement  of  The  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Paul  G.  Kaminski  before  the 
Subcommittee  on  Defense  Technology”,  Acquisition  and  Industrial  Base  of  the  Senate  Committee  on  Armed 
Services  on  Dual  Use  Technology.  May  17, 1995 

59  G.  Dean  Clubb,  “Blinding  Speed  Equals  Competitive  Advantage.”  Acquisition  Review  Quarterly.  Fall  1996 


Chapter  3 


The  Air  Force  Product  Development  Process 

This  chapter  provides  an  overview  of  the  various  organizations,  stages,  and  processes 
involved  in  the  current  Air  Force  product  development  process.  It  is  intended  to  provide  an 
understanding  of  the  actors  and  their  roles  in  the  process  of  developing  a  procurement  schedule, 
which  is  described  in  later  chapters.  The  description  provided  here  is  quite  general,  and  any 
individual  development  project  may  follow  a  slightly  different  path. 

The  description  is  based  on  federal,  DoD,  and  Air  Force  regulations  and  instructions, 
published  books  on  the  defense  acquisition  process,  material  for  acquisition  training  courses 
from  industry  and  specific  companies,  interviews  and  discussions  with  a  wide  range  of  people, 
and  personal  experience  within  the  acquisition  process.  Appendix  3  contains  references  as  well 
as  additional  detail  on  each  process. 


A.  Core  Product  Development  Processes 

Air  Force  product  development  processes  can  be  separated  into  six  distinct  stages: 
identifying  the  need,  developing  the  requirements,  allocating  resources,  planning  the  acquisition, 
contracting,  developing  the  product  and  establishing  the  process  to  produce  it. 
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Force  planning  determines  which  systems  are  needed.  Requirements  determine  what  the 
new  system  must  do  to  meet  the  need.  Resource  allocation  determines  the  funding  for  the 
development  effort,  given  the  range  of  activities  and  responsibilities  of  the  services.  Acquisition 
planning  entails  determining  how  the  system  will  be  contracted  and  creating  the  plan  to  develop 
the  system.  Contracting  entails  selecting  the  contractor  to  develop  the  product  and  specifies  the 
contract  conditions.  Development  turns  the  ideas,  requirements,  resources,  and  plans  into  a 
working  system  that  can  be  produced. 


Technological  Opportunity 
Emerging  Threat 
Current  Deficiency 
Changed  Military  Strategy 


First  Initial 
Production  Operational 
Item  Capability 


Figure  3-1:  Major  Steps  in  the  Air  Force  Product  Development  Process 

Identification  of  need  begins  when  an  operational  deficiency,  an  emerging  threat,  a 
technological  opportunity,  or  a  change  in  military  strategy  occurs.  Influences  at  this  stage 
include  the  Pentagon’s  long-range  planning,  the  Major  Commands’  modernization  planning 
process,  and  priorities  of  the  senior  leadership.  A  “mission  needs  statement”  specifically 
identifies  the  need,  which  leads  to  a  Milestone  0  decision  authorizing  further  analysis  and 
determination  of  system  requirements.  To  determine  those  requirements,  the  major  commands 
provide  an  analysis  and  obtain  the  Pentagon’s  approval.  The  result  is  an  “operational 
requirements  document”  outlining  what  a  new  system  must  do. 

The  major  commands’  resource  planning  process  and  the  Pentagon’s  programming, 
planning,  and  budgeting  systems  identify  the  dollars,  equipment,  and  number  of  people 
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authorized  for  the  project.  The  result  is  a  program  objective  memorandum  and  a  requested 
budget  sent  to  Congress.  The  program  offices  then  develop  an  acquisition  plan  and  obtain 
approval  within  the  Pentagon.  This  results  in  a  Milestone  I  decision  authorizing  contracting  and 
development  efforts. 

A  Program  Office  then  produces  a  request  for  proposal  (RFP)  and  selects  the  winning 
proposal  from  those  submitted  by  contractors.  The  result  is  a  contract  that  describes  the 
specifications  for  the  project  and  the  terms  under  which  the  contractor  will  pursue  it. 

The  contractor  then  designs  both  the  system  and  the  process  used  to  produce  it.  The 
program  office  oversees  this  process  and  the  Pentagon  oversees  the  Program  Office.  Completion 
of  development  is  marked  by  delivery  of  the  first  production  item.  Each  of  these  major  areas  will 
be  described  in  more  detail  later  in  the  chapter. 

The  acquisition  process  typically  proceeds  through  formal  phases  and  milestones 
designed  to  allow  for  periodic  review.  Need  identification  occurs  in  the  Pre-Milestone  0  phase. 
Setting  requirements,  allocating  resources,  performing  early  acquisition  planning  occur  during 
phase  0.  The  contracting  and  the  development  effort  occur  during  phases  I  and  II.  The 
development  effort  is  typically  complete  by  Milestone  HI  or  the  production  decision  that  marks 
the  beginning  of  Phase  HI.  Milestone  II  and  Milestone  HI  decisions  are  administrative-based 
decisions  that  do  not  always  tie  directly  to  event-based  milestones  in  the  development  process. 
Figure  3-2  below  shows  the  relationship  between  the  process  areas  and  the  milestone  phases. 

An  overall  metric  of  this  process  is  acquisition  response  time:  the  time  from  when  the 
need  arises  to  the  time  when  the  system  is  fielded  and  ready  for  use.  Development  time  is  the 
major  component  of  acquisition  response  time. 

Understanding  how  this  process  complex  requires  a  more  detailed  look  at  the 
organizations  and  sub-processes  involved. 
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Figure  3-2:  Major  Steps  in  the  Air  Force  Product  Development  Process 


B.  Organizations  Involved  in  Development 

The  organizations  primarily  involved  in  developing  defense  systems  are  the  users,  the 
service  headquarters,  the  Program  Offices,  and  the  defense  contractors.  Each  group  plays  a 
different  role  during  various  stages,  and  still  more  organizations  play  secondary  roles.  Figure  3-3 
shows  the  distribution  of  various  sub-processes  by  the  organizations  primarily  responsible  for 
them 

The  users-the  ultimate  customers-do  much  of  the  planning  and  establishing  of 
requirements.  The  users  are  organized  by  different  Air  Force  mission  areas  into  major 
commands  such  as  Air  Combat  Command  (ACC),  Air  Mobility  Command  (AMC),  and  Air 
Force  Space  Command  (AFSPC);  or  by  geographic  area  such  as  US  Air  Force  Europe  (USAFE) 
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and  Pacific  Air  Forces  (PACAF).  A  major  command’s  mission  is  to  organize,  train,  equip,  and 
maintain  combat-ready  forces  for  use  by  the  Unified  commands,  such  as  US  Central  Command, 
which  are  composed  of  units  of  all  the  services. 

Each  of  the  major  command  headquarters  contains  planning  offices  that  project  future 
needs,  requirements  officers  who  identify  future  requirements  for  systems,  and  programming 
offices  that  project  the  budget  for  major  command  activities.  (These  roles  will  be  discussed  in 
detail  below.)  The  officers  in  these  positions  are  typically  from  the  operational  units  within  the 
major  command. 
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Figure  3-3:  Major  Product  Development  Processes  by  Organization 
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The  service  headquarters  and  the  Department  of  Defense  together  compose  the  Pentagon. 
Each  service  headquarters  is  the  approval  authority  for  planning,  requirements,  and  acquisition, 
allocates  resources,  and  oversees  the  development  effort.  Each  function  is  duplicated  at  the  DoD 
level,  with  the  Office  of  the  Secretary  of  Defense  and  the  Joint  Chiefs  of  Staff  playing  an 
integrating,  authorizing,  and  oversight  role.  DoD  typically  becomes  involved  at  this  level  only 
with  the  largest  defense  acquisition  programs  and  those  with  joint  service  application. 

Many  functional  and  cross-functional  organizations  fulfill  the  service  headquarters’  role 
in  the  development  effort.  Those  organizations  are  divided  between  the  secretariat  and  air  staff. 
The  key  organizations  include  the  deputy  chief  of  staff  for  plans  and  programs,  the  assistant 
secretary  for  acquisition,  the  assistant  secretary  for  financial  management,  and  the  director  for 
program  integration.  Headquarters-level  groups  such  as  personnel,  manpower,  logistics,  C4I, 
and  test  also  become  involved  through  several  organizations  such  as  the  Working  and 
Overarching  Integrated  Product  Teams  and  the  Requirements  Review  Councils.  These 
organizations  are  the  primary  interface  with  senior  Air  Force  leaders,  DoD,  and  external 
organizations  such  as  Congress  and  the  administration. 

The  System  Program  Offices  oversees  planning  and  contracting  for  major  weapon 
systems  or  groups  of  similar  weapons,  and  act  as  the  primary  interface  with  the  contractor 
community..  Program  Offices  are  supported  by  Development  Centers  and  Logistics  Centers, 
which  provide  the  necessary  personnel.  Each  Program  Office  typically  oversees  many  product 
development  efforts. 

Before  developing  a  new  product,  defense  contractors  must  win  the  contract.  This 
involves  several  organizations  within  the  company:  marketing,  typically  referred  to  as  the 
business  development  group;  the  proposal  development  team,  and  the  integrated  product  team, 
which  develops  the  actual  product  and  processes.  Financial  management,  engineering,  and 
manufacturing  groups  oversee  the  company’s  development  efforts. 

Many  other  entities  such  as  the  testing  community,  the  defense  laboratories,  and  various 
defense  think  tanks  also  influence  a  development  project  but  play  a  secondary.  Higher-level 
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decision  makers  such  as  the  administration  and  Congress  are  primarily  involved  in  funding 
decisions  for  major  development  efforts  and  typically  not  in  the  details. 


C.  Identification  of  Needs 

In  the  Air  Force,  the  long-range  planning  group  within  headquarters  Air  Staff,  the 
modernization  planning  process  through  the  major  commands,  and  an  ad  hoc  process  based  on 
direction  from  senior  leadership  identify  current  and  future  needs.  Needs  based  on  current  and 
future  threats,  military  strategy,  current  military  capabilities,  and  available  technology  are 
documented  and  approved  through  the  mission  needs  statement,  which  in  turn  feeds  the 
requirements  and  resource-allocation  processes. 


Threat  Strategy 
Capabilities 


Figure  3-4:  Components  of  Identifying  Needs  and  Force  Planning 
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C.l  Long-Range  Planning 

The  Pentagon’s  Strategic  Planning  Office  was  established  in  1996  to  focus  on  a  25-to-40 
year  time  frame.  This  new  organization  has  not  yet  had  a  significant  impact  on  product 
development  but  may  in  the  future. 


C.2  Air  Force  Modernization  Planning  Process 

The  modernization  planning  process  determines  Air  Force  needs  for  new  or  improved 
capabilities  to  ensure  that  the  service  can  accomplish  its  mission,  including  the  president’s 
national  security  strategy  and  national  military  strategy.  The  process  used  to  convert  the  national 
military  strategy  into  the  required  weapon  systems  is  referred  to  as  “strategy-to-task."  Once  the 
tasks  are  defined,  WHO  evaluates  the  capabilities  of  existing  forces  and  identifies  requirements 
for  new  and  upgraded  systems.  This  —  process  referred  to  as  “task-to-need."  (The  processes  used 
to  accomplish  the  “strategy  to  task”  and  the  “task  to  need”  analyses  are  defined  in  several 
documents,  including  the  CJCS  MOP  77  Requirements  Generation  System  Policy  and 
Procedures ,  Air  Force  Directive  AFPD  10-14  Modernization  Planning,  and  Air  Force 
Instruction  AFI 10-1401  Modernization  Planning  Documentation.) 
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Modernization  Planning  Process 

•Perceived  Threat  Leadership  Input 

•Current  Capabilities 
•Leadership  Desires 
•Internal  Ideas 
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Figure  3-5:  The  Modernization  Planning  Process 


The  Air  Force  modernization  planning  process  is  conducted  by  the  major  commands 
through  a  number  of  mission  area  teams  and  associated  technical  planning  integrated  product 
teams  (TPIPTs).  These  plans  project  25  years  into  the  future  and  guide  investments  by  the 
scientific,  development,  and  contractor  communities.  The  major  commands  lead  the  mission 
area  teams,  and  the  TPIPTs  are  managed  by  Air  Force  Materiel  Command  and  the  teams’ 
associated  product  centers. 

Thirty-eight  mission  area  teams  and  functional  area  teams  analyze  capabilities  in  specific 
mission  areas.  One  example  is  the  Aerospace  Control  Mission  Area  Team,  which  examines 
offensive  counter-air,  defensive  counter-air,  and  theater  missile  defense.  (The  appendix  contains 
more  detailed  information  on  these  teams  and  their  makeup.)  The  teams  are  supported  by  sub¬ 
teams  associated  with  each  weapon  system,  and  by  technical  planning  integrated  product  teams. 
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The  TPIPTs  include  representatives  from  Program  Offices,  defense  laboratories,  and 
industry.  These  teams  identify  potential  systems  and  concepts  to  meet  needs  set  by  the  mission 
area  teams.  The  TPIPTs  also  identify  future  technological  needs  that  direct  much  Air  Force 
laboratory  research. 

All  these  teams  follow  the  “strategy-to-task”  and  the  “task-to-need”  process  to  determine 
current  capabilities,  future  deficiencies,  and  how  best  to  overcome  those  deficiencies.  After 
estimating  potential  enemy  forces  and  capabilities  for  25  years  and  projecting  Air  Force 
requirements,  mission  area  assessments  and  needs  analysis  rely  on  the  national  military  strategy 
to  determine  future  needs.  Mission  solution  analysis  then  determines  the  major  commands’ 
preferred  solutions  for  correcting  deficiencies. 

Mission  Area  Assessment  (MAA)  analyzes  the  ability  of  the  unified  commands  and  the  major 
commands  to  undertake  their  assigned  tasks  and  to  fulfill  a  military  contingency  plan. 

Mission  Needs  Analysis  (MNA)  then  uses  as  the  task-to-need  process  to  determine  if  existing 
forces  can  meet  the  current  and  future  assigned  missions.  Identified  mission  needs  are 
documented  in  a  mission  needs  statement,  which  is  used  to  make  a  Milestone  0  decision  for 
developing  any  new  or  modified  system. 

Mission  Solution  Analysis  (MSA)  evaluates  potential  solutions  and  produces  an  “unconstrained 
set  of  preferred  options.”"  These  options  may  include  changes  in  operations  tempo,  readiness, 
training  procedures  and  tactics,  modification  programs,  force  structure  changes,  new 
acquisitions,  and  science  and  technology  programs. 

Mission  Area  Plans  (MAP)  use  the  results  of  the  mission  area  assessment  and  the  mission  needs 
analysis  to  “document  the  most  cost  effective  means  of  correcting  task  deficiencies  from  among 
nonmaterial  solutions,  changes  in  force  structure,  system  modification  or  upgrades,  science  and 
technology  applications,  and  new  acquisitions.”61  The  MAPs  are  roadmaps  outlining  the 


60  Briefing  by  HQ  USAF/XOXP,  AQ/XO  Offsite.  1997. 

61  AFI  10-1401  Modernization  Planning  Documentation. 
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modernization  plan  for  each  weapon  system  in  the  mission  area.  Function  area  plans  are 
developed  for  cross-cutting  areas  such  as  communications. 

The  modernization  planning  process  culminates  with  a  four  star  review  by  senior  Air 
Force  leaders,  who  approve  the  mission  area  and  functional  area  plans.  The  potential  solutions  in 
the  MAPs  are  then  “racked  and  stacked”  to  determine  which  will  be  pursued. 

From  the  finalized  MAPs,  the  major  commands  develop  a  prioritized  list  of  new  or 
modified  systems.  Requirements  for  new  systems  and  modification  of  existing  systems 
identified  through  this  process  will  help  determine  how  much  funding  each  service  receives,  as 
well  as  when  the  process  for  establishing  specific  requirements  begins  and  pre-acquisition 
planning  occurs. 


C.3.  Leadership-Directed  Projects 

In  a  directed  project,  a  senior  leader  begins  a  development  effort  not  included  in  mission 
area  plans.  This  frequently  used  top-down  process  contrasts  with  the  bottom-up  process  used  in 
modernization  planning.  Respondents  to  the  surveys  conducted  as  part  of  this  research  indicate 
that  42  percent  of  current  Air  Force  projects  were  initiated  as  the  result  of  leadership  direction  as 
opposed  to  formal  modernization  planning.*0 

Directed  programs  may  result  from  a  new  technology,  a  highly  visible  deficiency,  or  the 
personal  interest  of  a  national  or  service  leader.  These  directed  programs  will  be  incorporated 
into  future  mission-area  plans  for  the  major  commands.  A  mission  needs  statement  or 
operational  requirement  document  used  for  this  kind  of  program  often  references  the  senior 
leader’s  direction. 


62  The  surveys  and  responses  will  be  discussed  in  detail  in  Chapters  7-11. 
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C.4.  Mission  Needs  Process 

A  mission  needs  statement  (MNS)  documents  the  capability  required  to  accomplish  a 
certain  operational  task,  as  well  as  the  inability  to  fulfill  the  need  through  training,  tactics,  or 
other  non-materiel  solutions.  A  validated  MNS  is  required  for  a  Milestone  0  decision,  which 
allows  for  early  studies  of  alternatives  and  the  operational  and  cost  implications  of  developing  a 
new  system. 

Generating  the  Mission  Needs  Statement 

The  major  commands  write  the  mission  needs  statement  based  on  the  outcome  of  the 
mission  needs  analysis  and  mission  solution  analysis.  A  major  command  will  also  generate  a 
mission  needs  statement  at  the  direction  of  senior  leaders.  Major  commands  submit  the  mission 
needs  statement  to  service  headquarters  for  validation. 


Figure  3-6:  Mission  Needs  Statement  Generation  and  Validation 


3.  The  Air  Force  Product  Development  Process 


81 


Validating  the  Mission  Needs  Statement 

The  validation  process  ensures  that  all  operational  needs  are  properly  documented  and 
agreed  to  by  vkious  functional  groups  before  any  development  effort  is  begun. 

A  major  command  forwards  a  completed  MNS  to  the  Operational  Requirements  Division  of  the 
Pentagon,  where  it  is  assigned  to  a  requirements  action  officer  who  shepherds  it  through  the 
validation  process.  This  officer  submits  the  MNS  to  the  working  level  of  the  Requirements 
Review  Council,  which  consists  of  major  and  lieutenant  colonel-level  action  officers  from  the 
functional  groups.  Following  approval  at  the  working  level,  the  MNS  is  reviewed  by  the 
Requirements  Review  Council,  which  consists  of  colonel  and  brigadier  general-level  officers 
from  various  functional  groups.  Following  council  approval,  the  MNS  is  then  submitted  to  the 
Air  Force  Requirements  Oversight  Council  (AFROC),  a  two-star-general-level  review  by  the 
deputy  chiefs  of  staffs. 

Once  reviewed  and  approved  by  the  AFROC,  the  MNS  is  returned  to  the  major  command 
for  final  review  and  signature  by  the  commander.  The  final  document  is  then  sent  to  the  Air 
Force  chief  of  staff  for  approval  and  signature.  Depending  on  the  size  of  the  potential  project  or 
the  joint  applicability,  the  MNS  may  also  then  be  sent  to  the  Joint  Requirement  Oversight 
Council  (JROC)  within  the  Joint  Chiefs  of  Staff  for  further  validation.  The  MNS  is  officially 
validated  by  the  Air  Force  chief  of  staff  or  by  the  Joint  Requirements  Oversight  Council. 

The  completed  mission  need  statement  is  then  sent  to  the  Assistant  Secretary  of  the  Air 
Force  for  Acquisition,  or  to  the  undersecretary  of  defense  for  acquisition  and  technology.  That 
person  makes  the  Milestone  0  decision  to  authorize  studies  of  how  to  best  meet  the  identified 
need  and  to  define  the  operational  requirement  for  the  new  or  modified  system. 
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D.  Generating  and  Validating  Requirements 

Requirements  document  the  capabilities  a  new  system  must  have  to  succeed.  In  practice, 
there  is  significant  overlap  between  the  process  of  establishing  requirements  and  the  planning 
activities  described  earlier.  The  requirements  community  consists  of  groups  at  the  major 
command  headquarters  and  the  Pentagon.  Each  major  command  has  a  director  for  requirements 
who  is  responsible  for  evaluating  current  capabilities,  identifying  deficiencies,  and  defining  the 
requirements  for  new  systems  in  operation  requirements  documents.  Within  Air  Force 
headquarters,  the  Operational  Requirements  Division  (HQ  AF/XOR)  reviews  the  documents, 
coordinates  them  with  other  Pentagon  organizations,  and  maintains  the  library  of  validated 
documents.  Depending  on  the  project’s  size  and  scope  or  joint  service  applicability,  the 
document  must  be  validated  by  the  Air  Force  chief  of  staff,  the  Air  Force  Requirements 
Oversight  Council  (AFROC)  or  the  Joint  Requirements  Oversight  Council  (JROC). 


Figure  3-7:  The  Process  for  Generating  and  Approving  Requirements 
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D.l.  Generating  Requirements 

Operational  deficiencies  and  solutions  are  turned  into  a  set  of  requirements  that  form  the 
basis  of  a  development  effort.  When  complete,  the  systems  will  be  tested  to  determine  how  well 
they  meet  the  requirements.  The  major  commands  specify  requirements  by  evaluating 
alternatives,  performing  Cost  and  Operational  Effectiveness  Analyses  (COEAs),  determining  the 
operating  parameters  for  a  specific  system,  and  developing  the  operational  requirements 
document. 

The  general  characteristics  of  the  desired  system  are  defined  by  various  outputs  from 
modernization  planning-often  the  mission  solution  analysis-or  by  directions  from  senior 
leadership.  The  process  focuses  on  how  to  fulfill  the  need  with  the  selected  system,  how  the 
system  should  work,  and  what  the  specific  operating  characteristics  should  be.  For  example,  if 
service  leaders  have  selected  a  new  aircraft  to  fill  a  specific  need,  the  requirements  will  define 
what  the  characteristics  of  the  new  aircraft  must  be  to  fill  the  need. 

The  primary  steps  involved  in  generating  requirements  are  analyzing  alternatives, 
analyzing  cost  and  operational  effectiveness,  and  generating  the  operational  requirements 
document. 

Analysis  of  alternatives  (AoA)  identifies  alternative  methods  of  meeting  the  operational 
requirements  and  resolving  mission  deficiencies  following  a  Milestone  0  decision.  The  results  of 
these  concept  studies  are  used  to  prepare  the  cost  and  operational  effectiveness  analysis. 

Cost  and  operational  effectiveness  analysis  (COEA)  is  used  to  assist  decision  makers  in 
selecting  the  most  cost-effective  method  to  fulfill  a  mission  need.  The  COEA  process  compares 
several  solutions  on  the  basis  of  cost  and  operational  effectiveness  and  documents  the  rationale 
for  choosing  the  preferred  solution.  COEAs  are  required  for  all  major  defense  acquisition 
programs.  Formal  COEAs  are  not  required  and  are  not  typically  performed  for  smaller  programs 
(ACAT  H  and  HI). 

The  Operational  Requirements  Document  (ORD)  describes  the  concept  of  operations  and  the 
specific  performance  measures — both  the  minimum  and  threshold  (desired)  levels  that  the  new 
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system  is  to  fulfill.  If  the  system  cannot  meet  the  threshold  parameters,  senior  leaders  must 
decide  whether  to  continue  the  project.  These  key  performance  parameters  are  included  in  the 
acquisition  program  baseline  and  are  reviewed  as  part  of  the  milestone  decision  points.  No 
acquisition  effort  can  proceed  through  Milestone  I  without  a  validated  operational  requirements 
document. 

The  ORD  is  updated  before  full-scale  development  (Milestone  II)  and  production 
(Milestone  HI)  to  include  additional  information  as  the  system  is  further  defined.  A  requirements 
correlation  matrix  tracks  changes  in  the  requirements  over  time  and  documents  the  reasons  for 
the  changes.  These  and  other  analyses  form  the  basis  for  the  subsequent  acquisition  efforts  for 
the  system. 

D.2.  Approving  the  Requirements 

Leaders  in  the  major  commands  review  the  ORD,  which  is  then  submitted  to  headquarters 
and  the  Requirements  Review  Council  and  follows  a  path  similar  to  that  of  the  Mission  Needs 
Statements.  Validating  the  requirements  can  take  several  months  to  several  years,  depending  on 
the  contentiousness  of  the  issues  and  the  desires  of  the  leadership. 

A  validated  ORD  is  considered  the  definitive  statement  of  users’  requirements  and 
becomes  the  technical  input  into  the  acquisition  process.  Once  the  requirements  are  established 
and  validated  through  this  process,  they  cannot  be  changed  without  a  new  review.  A  validated 
ORD  is  required  to  authorize  the  start  of  a  formal  acquisition  process. 


E.  Allocating  Resources 

Following  the  modernization  planning  process  and  concurrent  with  the  requirements 
process,  any  acquisition  effort  must  successfully  negotiate  and  obtain  funding,  equipment,  and 
personnel  through  the  resource  allocation  process.  This  process  is  used  to  develop  Air  Force  and 
DoD  budgets  for  all  activities,  and  includes  the  major  commands,  the  Pentagon,  DoD,  the 
administration,  and  Congress.  Several  different  functions  are  involved,  including  programming, 
which  authorizes  activities;  budgeting,  which  projects  costs  and  develops  annual  budget  requests; 
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and  resource  distribution,  which  disburses  and  tracks  funds  once  Congress  appropriates  them. 
Approved  and  scheduled  funding  for  development  and  production  is  required  for  the  Milestone  I 
decision  that  authorizes  the  start  of  a  project.  , 


Resource  Allocation  Process  and  PPBS 


Congress 

Defense 

Authorization 

Request 


Congress 

I  President’s 
Budget 
-  1  Year 
funding 


Resource  allocation  and  the  Pentagon’s  programming,  planning,  and  budgeting  system 
(PPBS)  include  multiple  steps  over  a  two-year  period.  Each  major  command  develops  a  Program 
Objective  Memorandum  (POM),  which  the  Pentagon  then  uses  to  develop  a  POM  for  each  service 
and  the  total  annual  request  included  in  the  president’s  budget.  Once  Congress  has  allocated  the 
money,  funds  are  distributed  in  accordance  with  the  POM  as  defined  by  defense  appropriations 
law. 
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E.l.  Developing  the  Program  Objective  Memorandum 

Every  year  each  major  command  projects  the  resources  required  to  accomplish  the 
missions  it  must  fulfill  and  to  procure  the  new  systems  it  desires.  This  information  is  captured  in 
the  program  objective  memorandum  (POM),  which  covers  activities  ranging  from  purchasing 
fuel  and  maintaining  systems,  to  developing  new  products,  to  training  and  deploying  troops  and 
sustaining  existing  operations.  The  POM  covers  all  funding  projections  for  six  years  and  force 
structure  for  nine  years,  and  is  submitted  two  years  before  the  first  fiscal  year  to  which  it 
pertains.  Headquarters  for  the  major  command  typically  develops  this  document  with  input  from 
each  group  within  the  command.  After  internal  reviews,  the  commander  approves  the  request 
and  it  is  forwarded  to  service  headquarters.  This  document  then  becomes  the  initial  input  to  the 
Pentagon’s  planning,  programming,  and  budgeting  system. 


E.2.  The  Pentagon’s  Planning,  Programming,  and  Budgeting  System 

The  PPBS  is  a  multi-step  annual  process  used  to  authorize  activities,  allocate  resources, 
and  develop  the  service  and  DoD  budgets.  Secretary  George  MacNamara  began  the  PPBS 
system  during  the  1960s  in  an  attempt  to  bring  focus,  coordination,  and  control  to  the  defense 
planning  and  budgeting  processes. 

Each  activity  the  service  undertakes  must  be  covered  by  a  program  element,  which 
allocates  all  funds  and  resources.  The  lowest  level  in  the  PPBS  system,  the  program  element, 
can  represent  a  single  weapon  system  undergoing  development  or  an  ongoing  military  operation. 
A  Program  element  monitor  (PEM)  is  the  official  spokesperson  for  that  activity  within  the 
Pentagon. 

Planning  and  Programming 

The  planning  and  programming  process  develops  and  integrates  the  Program  Objective 
Memorandum  and  maintains  the  future  year  defense  plan  (FYDP).  These  documents  authorize 
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the  services  to  both  carry  out  and  budget  each  specific  activity.  The  director  of  program  and 
evaluation  under  the  Air  Force  chief  of  staff  oversees  the  planning  and  programming  process.63 

The  planning  function  within  the  Pentagon  begins  with  the  submission  of  the  POMs  from 
the  major  commands.  These  inputs  assist  the  chiefs  of  staff  and  the  service  secretaries  in 
advocating  their  positions  and  developing  Defense  Planning  Guidance  (DPG).  The  latter  is  a 
high-level  document  that  sets  the  stage  for  apportioning  resources  among  the  services  and  within 
various  mission  areas.  After  the  DPG  is  issued,  the  major  commands  update  their  POMs  based 
on  the  expected  allocations  among  services  and  mission  areas. 

The  programming  function  takes  the  updated  POMs  sand  develops  a  consolidated 
program  outline  of  all  Air  Force  activities.  This  program  is  then  worked  through  an  elaborate 
process  to  ensure  that  it  fits  within  the  expected  budget  and  program  guidelines  directed  by 
Office  of  the  Secretary  of  Defense,  and  that  it  meets  the  requirements  of  the  senior  Air  Force 
leadership.  The  elaborate  process  used  to  make  these  decisions  and  tradeoffs  is  known  as  the 
enhanced  Air  Force  corporate  process,  and  includes  over  70  separate  weapon  teams,  10  mission 
area  panels,  and  3  additional  levels  of  review;  before  it  is  submitted  and  approved  by  the  Chief  of 
Staff  and  the  Secretary  of  the  Air  Force.  This  entire  process  involves  the  participation  of  a 
significant  number  of  people  in  the  Pentagon. 

The  secretary  of  defense  then  approves  a  program  decision  memorandum,  which  outlines 
the  proposed  programs  for  the  services.  This  memorandum  forms  the  basis  for  the  future  year 
defense  plan  (FYDP),  which  outlines  the  path  for  the  military  services  over  the  next  six  years. 
The  program  decision  memorandum  is  then  submitted  to  committees  in  Congress  for  their 
authorization. 

The  Enhanced  Air  Force  Corporate  Process 

The  enhanced  Air  Force  corporate  process  includes  five  levels  of  review  within  the 
Pentagon  prior  to  the  final  review  by  the  chief  of  staff  and  the  secretary  of  the  air  force.  It  is 
intended  to  provide  the  senior  leadership  with  the  “corporate  position  while  retaining  the 


63  HQ  USAF/PE  was  recently  placed  under  HQ  USAF/XO  to  create  HQ  USAF/XOP 
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responsibilities  of  the  functional  organizations.”  In  other  words,  the  “corporate  position”  is  a 
negotiated  result  and  compromise  solution  on  the  funding  levels  for  various  Air  Force  activities 
between  the  organizations  and  personalities  represented  within  the  various  panels,  board,  groups, 
and  council. 

$ 

The  different  levels  of  review  are  described  below  and  depicted  in  Figure  3-9.  In  1997, 

£ 

development  and  procurement  activities  represented  32  percent  of  the  total  Air  Force  budget. 

Integrated  process  teams  (IPT)  are  staffed  by  the  various  program  element  monitors  and  action 
offices  associated  with  a  weapon  system.  The  IPT  is  the  single  point  of  contact  for  the  major 
commands  to  specific  programs.  There  are  roughly  70  separate  weapon  system  IPTs. 

Mission  and  mission  support  panels  serve  as  the  Air  Force  “centers  of  expertise”  and  represent 
the  first  level  of  corporate  review  within  each  of  10  mission  areas.  While  retaining  a  corporate  or 
Air  Force  wide  perspective,  the  panels  must  still  play  the  role  of  advocate  for  a  particular  project 
within  the  corporate  process. 

The  Air  Force  Group  undertakes  the  first  integrated  Air  Force-wide  review  and  consists  of  23 
core  members  and  7  members  representing  the  functional  areas  of  the  Air  Force.  During  the 
POM  process,  this  group  is  empowered  with  “off-the-table”  decision  authority  (They  can  say  no 
to  a  decision,  but  not  yes)  as  it  brings  ideas  forward  to  the  Air  Force  Board. 

The  Air  Force  Board  is  a  two-star-level  review  staffed  by  the  deputies  of  senior  leaders  in  each 
major  functional  area.  The  board  now  consists  of  23  core  functional  areas  and  7  other  members. 

The  Air  Force  Council,  a  three  and  four-star-level  review,  consists  of  senior  leaders  from  the  various 
functional  areas  and  major  commands.  The  council  provides  recommendations  that  are  coordinated  at 
senior  levels  across  the  Air  Force  and  forwarded  to  the  Air  Force  chief  of  staff  and  the  secretary  of  the 
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Figure  3-9:  The  Enhanced  Air  Force  Corporate  Review  Process* 


64  Developed  from  USAF  Document  “The  Planning,  Programming,  And  Budgeting  System  &  The  Air  Force 
Corporate  Structure  (AFCS)  Primer,  9th  Edition.”  AF/XPPE.  May  1998 


90  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Budgeting 

The  Air  Force  budget  process  is  organized  and  controlled  primarily  by  the  assistant 
secretary  of  the  Air  Force  for  financial  management,  and  more  particularly  by  the  budget 
division.  The  annual  budget  is  developed  based  on  the  results  of  the  program  objective 
memorandum.  The  budgeting  process  develops  more  accurate  estimates  of  the  costs  of  executing 
approved  activities  by  updating  previous  estimates  using  current  prices,  inflation  estimates,  and 
economic  forecasts.  The  budgeting  process  also  separates  the  required  funds  into  categories  used 
to  submit  the  overall  Air  Force  budget  estimate  and  compose  the  president’s  budget  to  Congress. 

Based  on  these  estimates  and  changes  as  directed  by  guidance  from  the  secretary  of 
defense,  the  Air  Force  Corporate  Process  Structure  reworks  the  Air  Force  program  to  create  the 
budget  estimate  submission  (BES).  The  BES  represents  the  Air  Force  input  into  the  annual  DoD 
budget  request.  Following  a  review  by  the  Office  of  the  Secretary  of  Defense  and  the  Office  of 
Management  and  Budget,  the  secretary  of  defense  issues  a  president’s  budget  decision  (PBD). 
The  services  again  adjust  their  program  to  comply  with  the  PBD  and  finalize  the  Air  Force 
budget.  The  president’s  budget  is  then  submitted  to  Congress  for  approval  and  enactment. 

Once  the  budget  is  passed  by  Congress  and  signed  by  the  president,  the  budget  and 
comptroller  communities  distribute  funding  and  track  expenditures  to  ensure  that  no  money  is 
spent  that  is  not  allocated  and  appropriated  for  a  specific  project. 

The  outcomes  of  this  entire  process-Program  Objective  Memorandum  and  the 
Future  Year  Defense  Plan—  are  inputs  into  the  acquisition  processes  and  critical  to  a  product 
development  effort.  With  the  added  expectation  of  an  approved  Operational  Requirements 
Document,  the  steps  typically  associated  with  the  acquisition  processes  begin.  Participants  in  the 
acquisition  processes  take  the  users’  requirements  and  expected  resources  and  develop  an 
acquisition  plan,  select  a  contractor,  and  develop  the  product  and  the  manufacturing  process. 
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F.  Acquisition  Planning  and  Approval 

In  the  acquisition  approval  and  milestone  decision  processes,  the  service  headquarters  and 
service  acquisition  executive  review  the  proposed  plans  and  decide  if  the  project  can  proceed  to 
the  next  stage. 


Acquisition  Planning  and 
Approval  Process 


Figure  3-10:  Acquisition  Planning  and  Approval  Processes 


F.l  Acquisition  Planning 

This  process  develops  plans  based  on  the  users’  Operational  Requirements  Document. 

The  major  acquisition  planning  steps  include  developing  a  team,  collecting  information,  creating 
program  plans,  projecting  costs,  and  developing  an  acquisition  strategy.  The  major  products  are 
acquisition  plans,  cost  estimates,  proposed  project  schedules,  and  a  proposed  acquisition  program 
baseline  that  outlines  the  project’s  cost,  schedule,  and  performance  objectives. 
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The  planning  effort  is  accomplished  by  a  newly  formed  Program  Office  specific  to  the 
project,  or  by  a  team  within  an  existing  Program  Office.  The  Program  Office  typically  surveys 
potential  contractors  to  understand  abilities  and  technologies  available  to  the  project.  From  this 
information,  the  Program  Office  develops  a  series  of  plans  that  outline  how,  when,  and  at  what 
cost  the  project  can  proceed.  The  plans  typically  include  an  acquisition  plan,  a  source  selection 
plan,  and  system  engineering  plans  including  a  work  break  down  structure,  a  master  plan  and  a 
master  schedule.  The  plans  also  include  an  integrated  logistics  support  plan;  a  test  and 
evaluation  master  plan;  a  human  system  integration  plan;  and  a  threat  assessment  report.  These 
plans  and  reports  cover  the  entire  expected  life  of  the  program,  not  just  the  development  effort. 
During  this  phase,  the  Program  Office  develops  detailed  cost  estimates  showing  that  the  plans  are 
affordable  within  the  expected  resources,  as  required  for  a  Milestone  I  decision.  The  reports 
officially  required  for  a  Milestone  I  decision  for  major  defense  acquisition  programs  are  shown 
in  Table  3-1. 


Acquisition  Strategy  Report 

Acquisition  Program  Baseline 

Affordability  Assessment 

Future  Year  Defense  Plan  Funding  Profile 

Analysis  of  Alternatives 

Component  Cost  Analysis 

Cost  Analysis  Requirements  Description 


Description  of  Data  Requirements 
Exit  Criteria 

Independent  Estimate  of  Life  Cycle  Costs 
Operational  Requirements  Document 
Program  Office  Life  Cycle  Cost  Estimate 
System  Threat  Assessment  Report 
Test  and  Evaluation  Master  Plan 


Table  3-1:  Information  Required  for  a  Milestone  I  Decision  for  Major  Defense  Acquisition 
Programs 
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When  the  plans  are  complete,  the  Program  Office  holds  an  acquisition  strategy  review 
with  a  panel  of  outside  experts  and  consolidates  the  results  into  an  integrated  program  summary. 
This  summary  is  signed  by  the  program  director  and  forwarded  to  the  Pentagon  for  review  and 
approval. 

F.2.  Acquisition  Approval 

Milestone  approval  is  the  formal  process  used  to  review  projects  and  authorize  them  to 
proceed  to  the  next  acquisition  phase.  Formal  Milestone  decisions  are  required  by  Defense 
Acquisition  Regulation  5000.1,  which  gives  the  defense  acquisition  and  service  acquisition 
executives  milestone  decision  authority  for  major  programs.  For  smaller  programs  the  product 
center  commanders,  also  known  as  defense  acquisition  commanders  (DACs),  have  milestone 
decision  authority.  Milestone  decisions  are  intended  to  ensure  that  all  essential  issues  are 
addressed  prior  to  approval  for  the  program  to  proceed.  After  a  Milestone  I  decision  initiates  a 
formal  acquisition  program,  a  Milestone  II  decision  authorizes  full-scale  development,  and  a 
Milestone  El  decision  authorizes  full-rate  production. 

Before  a  milestone  decision,  a  project  must  pass  through  a  series  of  Pentagon  reviews. 
Each  major  organization  involved  with  any  aspect  of  the  development  program  is  represented  on 
two  respective  teams:  the  Working-Level  Integrated  Product  Team  (WIPT),  and  the  Overarching 
Integrated  Product  Team  (OIPT  The  people  and  organizations  involved  with  these  teams  overlap 
significantly  with  the  earlier  integrated  product  teams  that  allocated  resources. 

These  committees  are  intended  to  resolve  all  issues  before  submitting  the  entire  package 
to  the  Air  Force  System  Acquisition  Review  Council  and  the  assistant  secretary  of  the  Air  Force 
for  acquisition  for  final  approval.  The  largest  programs  are  reviewed  in  the  Office  of  the 
Secretary  of  Defense,  by  the  Joint  Requirements  Oversight  Council,  the  DoD-level  Overarching 
IPT,  and  the  Defense  Acquisition  Board  prior  to  approval  by  the  secretary  of  defense  for 
acquisition  and  technology. 

A  Milestone  I  Decision  yields  a  signed  acquisition  decision  memorandum  authorizing  the 
a  formal  acquisition  project  and  a  baseline  specifying  cost,  schedule,  and  performance. 
Deviations  from  the  approved  requirements  or  the  acquisition  strategy  require  similar  approval 
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from  the  milestone  decision  authority.  An  approved  Milestone  I  decision  allows  the  program 
office  to  start  the  contracting  processes. 


G.  Contracting 

Contracting  is  a  critical  part  of  the  development  process.  The  basic  contracting  process  is 
dictated  by  federal  acquisitions  regulations  and  applies  across  all  federal  agencies.  Its  two 
primary  purposes  are  to  select  a  contractor,  and  to  agree  on  terms  and  conditions  for  the  contract 
Contracting  can  be  separated  into  three  periods:  planning  for  release  of  the  request  for  proposal, 
the  period  in  which  the  contractors  develop  their  proposals  and  selection  of  a  contractor  and 
award  of  a  contract  by  the  Program  Office. 


Contracting  Processes 
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Figure  3-11:  The  Contracting  Process 
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G.l  Solicitation  Planning 

The  many-step  process  of  planning  the  solicitation  overlaps  significantly  and  is  often 
indistinguishable  from  some  of  the  early  acquisition  planning  processes.  One  early  step  is 
market  research  to  identify  potential  contractors.  The  solicitation  planning  stage  also  includes 
developing  the  acquisition  plan,  which  schedules  the  contracting  steps  and  specifies  the 
requirements  for  the  solicitation. 

This  phase  also  spells  out  the  process  to  be  used  to  select  the  winning  contractor, 
including  the  evaluation  criteria  and  the  specific  weights  assigned  to  each.  The  criteria  and  the 
relative  weighting  are  provided  to  the  contractors  as  part  of  the  request  for  proposals.  The  source 
selection  criteria  cannot  be  changed  after  the  request  for  proposals  is  issued,  to  prevent 
manipulation. 

A  major  part  of  solicitation  planning  is  developing  the  formal  document  that  tells 
contractors  what  the  government  requires  and  how  the  contractors  are  to  respond.  Contractors 
are  typically  asked  to  comment  on  the  RFP  while  it  is  in  draft,  to  identify  particularly  onerous  or 
costly  requirements. 

The  first  part  of  the  RFP  typically  includes  a  model  contract  and  the  procedures  and 
ground  rules  for  selection.  The  second  part  describes  the  procedures  and  ground  rules  to  be 
followed  in  the  proposal,  the  basis  for  selection,  and  the  specific  information  needed  to  make  the 
selection.  This  section  usually  asks  for  technical,  management,  and  price  or  cost  proposals 
detailing  various  aspects  of  a  contractor’s  proposed  development  effort.  All  Air  Force  RFPs  are 
reviewed  by  the  centralized  RFP  support  team  to  ensure  that  the  requests  do  not  tell  contractors 
how  to  do  things  but  instead  simply  states  the  objectives. 

As  part  of  solicitation  planning,  the  Program  Office  develops  detailed  cost  estimates  for 
the  proposed  contracts  it  is  requesting.  This  helps  ensure  that  the  Program  Office  is  not 
requesting  more  development  effort  than  it  can  afford,  and  is  used  to  select  and  negotiate  with 
the  contractors. 
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Each  step  in  the  solicitation  planning  process  undergoes  legal  review  to  ensure  that  it 
adheres  to  appropriate  laws  and  regulations.  The  final  review  is  a  business  release  that  allows  the 
RFP  to  be  issued.  The  solicitation  planning  phase  ends  when  the  Program  Office  releases  request 
for  proposal,  — after  which  no  informal  discussion  between  industry  and  the  Program  Office  is 
allowed,  to  prevent  favoritism.  The  next  process  entails  development  by  the  contractors  of  a 
proposal. 


G.2  Proposal  Development 

For  defense  contractors,  winning  the  source  selection  process  is  the  most  important  aspect 
of  their  business.  Because  a  single  contractor  usually  develops,  produces,  and  maintains  defense 
systems,  not  being  selected  at  any  point  in  the  process  will  often  eliminate  a  company  from  the 
entire  market.  Defense  contractors  therefore  place  great  emphasis  on  their  ability  to  develop 
proposals  and  their  responsiveness  to  customers.  The  following  description  is  based  on  a  number 
of  industry  guidelines  for  developing  proposals,  interviews  with  industry  participants  in  the  Lean 
Aircraft  Initiative,  and  a  course  run  by  the  Educational  Service  Institute  that  teaches  government 
personnel  on  source  selection  procedures.45 

Proposal  development  and  marketing  activities  begin  long  before  release  of  the  RFP — 
and  indeed  even  before  the  Program  Office  establishes  a  project.  Companies  often  propose 
solutions  to  needs  directly  to  users,  the  Pentagon,  and  the  Program  Offices  in  an  attempt  to 
initiate  contracts  they  can  easily  win.  One  primary  function  of  companies’  marketing  divisions  is 
to  identify  upcoming  projects  that  fit  within  the  company’s  product  line.  Marketing  divisions 
seek  information  such  as  the  budget  for  potential  projects,  their  sponsors  and  their  interests  and 
concerns,  and  the  concerns  of  the  Program  Office  leaders.  One  objective  is  to  slant  specifications 
and  source  selection  criteria  toward  the  company’s  approach.  Marketing  divisions  also  identify 
potential  competitors,  their  likely  approaches,  and  their  level  of  interest  in  the  project.  Early 


65  The  description  is  obtained  primarily  from  two  company  proposal  development  guides  for  which  documentation 
was  obtained.  The  companies  requested  not  to  be  identified.  Interviews  with  other  industry  representatives 


3.  The  Air  Force  Product  Development  Process 


97 


market  research  ensures  that  a  company  has  adequate  time  and  the  necessary  information  to 
develop  an  effective  proposal,  as  companies  often  submit  their  proposals  right  after  the  legal 
minimum  of  30  days  following  release  of  the  RFP. 

When  a  company  identifies  a  project  that  the  firm  stands  a  significant  chance  of  winning, 
it  establishes  a  proposal  development  team.  This  initial  team  evaluates  the  information  from 
marketing  and  develops  a  strategy  for  winning  the  contract.  After  consulting  with  engineering 
groups,  the  team  selects  the  best  technical  approach  and  develops  a  “capture  plan”  detailing  and 
coordinating  the  company’s  efforts  to  win  the  contract.  The  capture  plan  includes  a  marketing 
plan  and  a  proposal  development  plan.  The  marketing  plan  ensures  that  the  Program  Office  is 
fully  informed~and  hopefully  convinced— of  the  benefits  of  the  company’s  specific  approach. 
Marketing  efforts  also  ensure  that  the  proposal  addresses  Program  Office  concerns,  and  the 
contractor’s  internal  proposal  evaluation  ensures  that  the  proposal  has  met  all  RFP  requirements 
and  provides  specific  answers  to  fulfill  the  criteria.  The  capture  plan  is  presented  to  senior 
management  for  their  approval. 

According  to  industry  guidelines,  “the  proposal  is  primarily  a  selling  tool  designed  to 
stress  customer  objectives  and  customer  benefits,  while  stating  the  customer’s  problem  in  his 
terms  and  presenting  the  solution  in  a  clear  and  straight  forward  manner.”  The  guidelines  stress 
that  “the  proposal  is  not  the  place  to  tell  the  government  it  is  wrong. . .  [That]  is  not  a  winning 
strategy.”  Instead,  a  company  often  portrays  alternative  approaches  by  other  firms  as  inferior,  to 
make  the  strongest  case  for  its  approach.  Many  companies  also  create  groups  that  solely  prepare 
or  assist  in  preparing  proposals,  led  by  a  proposal  manager  and  a  proposal  development 
specialist.  The  proposal  manager  often  becomes  the  program  manager  if  the  company  wins  the 
contract. 

Proposals  are  typically  written  in  sections,  each  targeted  to  different  evaluators  on  the 
source  selection  team.  Those  elements,  which  typically  include  an  executive  summary,  the 
technical  approach,  the  program  plan,  and  the  management  plan,  is  expected  to  stand  -alone. 


supported  the  statements  contained  in  the  guides.  The  process  described  is  similar  to  the  process  outlined  in  an 
Educational  Services  Institute  course  on  source  selections. 
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The  program  plan  and  the  technical  approach  require  creation  of  a  work  breakdown  structure  and 
the  systems  engineering  management  plan. 

Companies  often  have  the  option  of  submitting  an  alternative  proposal  that  presents  a 
program  different  from  the  one  the  government  has  requested.  Such  proposals  may  not  meet  all 
the  government's  requirements  but  may  showcase  an  innovative  approach  to  meeting  users’ 
needs.  However,  according  to  the  Aeronautical  Systems  Center  Pre-Award  Support  Office  (a 
group  that  helps  many  program  offices  develop  requests  for  proposals  and  run  source  selections) 
contractors  take  this  route  only  rarely. 

After  drafting  and  evaluating  the  proposal,  company  management  sets  the  price  it  will 
bid.  Managers  base  this  decision  based  on  cost  estimates  developed  by  the  proposal  team, 
evaluation  of  potential  competitors  strategies,  and  the  importance  of  winning  the  project.  Once 
the  final  price  has  been  set,  final  proposal  production  occurs.  Submission  of  contractors' 
proposals  begins  the  source  selection  process. 

G.3  Source  Selection 

Before  beginning  source  selection,  a  Program  Office  a  plan  that  describes  the  process  to 
be  used  to  evaluate  the  proposals.  The  Source  Selection  Authority  approves  this  plan  before 
release  of  the  RFP. 

The  selection  process  typically  includes  a  Source  Selection  Evaluation  Board  (SSEB) 
composed  of  several  panels  that  evaluate  different  aspects  of  each  proposal;  a  Source  Selection 
Advisory  Council  (SSAC);  and  a  Source  Selection  Authority  (SSA).  The  SSEB  is  solely 
responsible  for  evaluating  the  proposals  against  the  set  standards-it  does  not  evaluate  different 
proposals  against  each  other.  The  Source  Selection  Advisory  Council,  composed  of  senior 
military  and  government  personnel,  reviews  the  SSEB  finding,  compares  the  proposals,  and 
considers  contractors’  past  performance.  This  committee  makes  a  recommendation  to  the  Source 
Selection  Authority,  which  then  selects  the  competing  contractors  with  which  DoD  will 
negotiate.  The  committee  can  pursue  a  contract  without  negotiations  by  choosing  one  of  the 
original  proposals  unmodified. 
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Negotiations  further  refine  and  clarify  the  contractors’  proposals  to  better  meet  the 
expectations  and  concerns  of  the  Program  Office.  Care  must  be  taken  to  provide  all  competing 
contractors  with  similar  information  but  not  to  share  proprietary  information.  Companies  then 
modify  their  proposals  and  develop  their  “Best  and  final  offer.”  During  this  period  a  proposed 
formal  contract  between  the  Program  Office  and  the  contractor,  specifying  which  aspects  will  be 
made  legally  binding  and  under  what  conditions,  is  written  and  prepared 

Using  the  contractor’s  best  offer  and  the  finalized  contract,  the  Source  Selection 
Authority  can  choose  either  the  best- value  proposal  or  the  lowest  cost  technically  acceptable 
proposal.  Decision  makers  may  ignore  the  scores,  re-score  the  proposals,  or  declare  differences 
insignificant,  but  the  selection  is  susceptible  to  protest  and  judicial  review.  While  protests  occur 
frequently,  few  are  upheld.  Of  the  47  protests  in  1995,  for  example,  only  2  were  upheld."  The 
Air  Force  often  goes  to  great  lengths  and  expense  to  ensure  that  the  source  selection  is  “fire¬ 
proof’  and  can  withstand  a  protest. 

Following  the  source  selection  decision,  the  Program  Office  can  award  the  contract  and 
development  can  begin. 


H.  Developing  the  Product  and  Process 

Development  of  the  actual  product  and  the  process  used  to  make  it  is  conducted  primarily 
by  the  contractors.  The  contractors’  development  activities  are  overseen  by  the  Program  Office, 
to  identify  problems  early  on.  The  Pentagon  oversees  the  activities  of  the  Program  Offices  and 
the  contractors,  to  alert  senior  leaders  to  problems.  A  process  is  also  available  to  modify  the 
contract  and  requirements  to  adapt  to  unforeseen  events. 

H.1  Product  and  Process  Development 

Contractors  themselves  can  determine  how  best  to  develop  a  product  and  the  process  used 
to  make  it,  as  long  as  they  remain  within  the  bounds  of  the  contract.  Contract  specifications 


“  Lt  Gen  Franklin’s  Briefing  to  Industry  Day  at  Electronic  Systems  Center.  Hanscom  AFB  MA.  March  1996. 
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contract  can  be  extensive  if  they  include  the  system  engineering  management  plan  and  work 
breakdown  structure.  A  company’s  development  efforts  typically  follow  standard  system 
engineering  approaches,  with  a  series  of  internal  reviews,  tests,  and  audits  as  a  design  matures. 
Many  of  the  company’s  activities  are  driven  by  program  oversight  requirements. 

Contractors  receive  progress  payments  throughout  development.  The  amount  is, 
determined  by  the  cost  work  breakdown  structure  and  its  packages  as  described  in  the  contract 
and  as  measured  through  the  company’s  cost/schedule  control  system  (C/SCS),  which  they  keep 
in  accordance  with  DoD  accounting  requirements.  Contractors’  requests  for  payment  are 
certified  by  the  Program  Office  and  sent  to  the  Defense  Finance  and  Accounting  Service. 

H.2  Development  Oversight 

The  development  oversight  process  involves  two  distinct  levels:  program  oversight  of  the 
defense  contractor,  and  Pentagon  oversight  of  the  Program  Office.  Each  is  intended  to  ensure 
that  the  proper  actions  are  taken,  and  that  the  development  of  the  desired  product  is  proceeding 
within  the  technical  and  cost  parameters. 

Program  Office  Oversight  of  the  Contractor 

To  ensure  progress  towards  completing  the  design,  and  a  design  that  will  meet  the 
requirements,  the  Program  Office  conducts  periodic  reviews  of  both  the  design  and  its  financial 
performance.  These  reviews  offer  insight  into  the  technical  aspects  of  the  development  process 
and  allow  for  early  identification  of  potential  problems.  Progress  is  measured  against  the 
company’s  planned  or  contracted  schedule. 

The  Program  Office  performs  financial  oversight  through  the  cost/schedule  control 
system,  which  formalizes  periodic  reports  indicating  the  funds  spent  by  the  contractor  and  the 
amount  of  work  accomplished.  From  these  reports  the  Program  Offices  can  determine  the 
budgeted  cost  of  the  work  performed,  the  actual  cost  of  the  work  performed,  the  budgeted  cost  of 
the  work  scheduled,  and  the  estimated  cost  at  completion.  These  are  used  to  determine  if  the 
contractor  is  meeting  the  schedule  and  cost  estimates  established  in  the  contract. 
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Pentagon  Oversight  of  the  Program  Offices 

Pentagon  oversight  of  the  Program  Office  and  the  contractor  is  more  limited  than 
Program  Office  oversight  of  contractors.  Pentagon  oversight  is  often  dependent  on  Program 
Office  reporting.  Quarterly  reports  are  due  from  all  major  defense  acquisition  programs  as  part 
of  the  defense  acquisition  executive  summary  (DAES),  completed  by  the  Office  of  the  Secretary 
of  Defense.  Smaller  programs  are  subject  to  reviews  by  their  respective  program  executive 
officer  or  product  center  commander.  (Each  Development  Center  and  Program  Executive  Office 
also  conducts  an  annual  portfolio  review  with  the  senior  acquisition  executive  to  review  the 
breadth  of  projects.) 

Test  results  and  completion  of  major  milestones  on  schedule  indicate  that  the  technical 
aspects  of  a  program  are  likely  under  control.  Missed  milestones  or  failed  tests  indicate  that 
there  may  be  problems  and  often  provoke  greater  Pentagon  scrutiny  and  oversight.  A  final 
operational  test  and  evaluation  (OT&E),  which  follows  completion  of  the  development  effort  and 
precedes  full-rate  production.  OT&E  tests  the  system  against  the  operational  requirements 
document  and  provides  an  independent  assessment  of  the  development  effort. 

H.3  Change 

During  development  effort  requirements  may  change,  the  contract  may  change,  and 
annual  funding  may  change.  These  changes  may  be  directed  by  the  Pentagon  or  the  users,  or 
they  may  come  from  the  contractors.  Changing  established  program  plans  is  difficult  for  both 
DoD  and  contractors. 

Revisions  in  the  annual  projected  budget  are  one  of  the  major  sources  of  change  in 
defense  programs.  When  the  allocated  budget  is  changed,  the  Program  Offices  must  adapt  to  the 
often  lower  level  of  funds,  and  the  contract  must  often  be  re-negotiated.  Additional  requirements 
based  on  changing  circumstances,  or  the  availability  of  newer  technology,  may  also  change 
users’  desired  capabilities. 

Bottom-up  changes  are  often  the  result  of  problems  within  a  development  effort.  The 
inability  to  meet  cost,  schedule,  or  technical  performance  requirements  often  force  a  contractor 
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and  a  Program  Office  to  suspend  requirements,  increase  funding,  or  to  delay  the  product.  These 
changes  often  require  a  significant  amount  of  effort  to  obtain  agreement  from  the  Pentagon 
organizations  that  must  review  and  approve  them.  Such  changes  must  follow  a  similar  path  as 
the  original  approval  process,  as  must  requests  for  additional  funding,  which  must  also  compete 
with  other  projects.  What’s  more,  changes  to  the  acquisition  program  baseline  require  a  similar 
level  of  review  as  the  original  milestone  decision  process.  Surmounting  these  hurdles  requires 
considerable  effort  and  attracts  greater  scrutiny  to  a  project. 


Chapter  Summary 

The  Air  Force  product  development  process  described  is  a  complicated  and  elaborate 
process.  The  major  players  in  the  development  process  include  the  major  commands,  the  various 
Pentagon  organizations,  the  program  offices,  and  the  defense  contractors.  The  earliest 
development  processes  includes  many  processes  from  the  modernization  planning  process,  to  the 
mission  needs  process,  to  the  requirements  generation  and  approval  processes.  The  planning, 
programming,  and  budgeting  system  as  implemented  by  the  Air  Force  through  its  corporate 
process  is  an  very  elaborate  process  encompassing  6  layers  of  review  multiple  times  per  year. 
The  acquisition  processes  include  the  acquisition  planning  and  approval  process,  the  contracting 
process  and  the  contractor  oversight  process.  Many  of  the  processes  involve  multiple  levels  of 
reviews  within  the  Pentagon  and  major  commands.  Each  of  these  processes  are  currently 
required  to  develop  a  project.  While  the  Air  Force  product  development  process  may  be  slightly 
more  complicated  than  other  services,  the  basic  steps  are  similar.  How  the  structure  of  this 
complicated  process  effects  the  development  time  of  individual  acquisition  efforts  and  how  it 
effects  the  development  process  as  a  whole  is  the  subject  of  this  investigation. 


Part  2 


Research  into  the  Process  Used  to  Develop 
_ Project  Schedules _ 

Part  2  Overview 

The  areas  associated  with  long  cycle  times  in  Air  Force  development  projects  are  too 
numerous  to  study  within  the  time  and  resources  available.  Initial  interviews  and  preliminary 
research  indicated  that  the  process  used  to  develop  project  schedules  was  likely  to  play  a 
significant  role  in  determining  the  time  to  develop  new  products.  Thus,  I  chose  to  focus  the 
research  on  the  process  used  to  develop  the  project  schedules  and  its  impact  on  the  resulting 
development  time. 

Chapter  4  provides  the  justification  for  focusing  on  the  process  used  to  develop  project 
schedules.  It  also  describes  previous  research  on  the  topic  and  the  limited  available  data. 
Chapter  5  describes  the  selection  of  the  research  approach  and  identifies  a  multilevel  survey  as 
the  appropriate  method  to  collect  the  required  data.  Chapter  6  describes  the  Pentagon,  Program 
Office,  and  Contractor  surveys  used  to  collect  the  necessary  data  on  development  of  the  initial 
project  schedules,  and  discusses  the  characteristics  of  the  317  projects  surveyed.  Chapter  7 
describes  the  process  used  to  analyze  and  present  the  data. 
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Chapter  4 


Narrowing  the  Research  Focus  to  the  Schedule 
Development  Process  and  Its  Impacts _ 


The  primary  goal  of  this  study  is  to  identify  the  key  factors  that  influence  product 
development  time,  and  that  keep  the  military  from  making  reductions  similar  to  those  achieved 
with  commercial  products.  This  chapter  provides  the  rationale  used  to  narrow  the  research  focus 
from  acquisition  response  time  to  product  development  time,  and  then  to  the  programmatic 
aspects  of  a  project.  To  do  so,  the  chapter  describes  the  conceptual  framework  used  to  separate 
different  aspects  of  a  project  that  may  affect  development  time. 

This  chapter  includes  initial  observations  of  the  schedule  development  process  based  on 
preliminary  research,  describes  the  relationship  between  the  project  schedule  and  development 
time,  and  poses  the  series  of  initial  questions  that  guided  the  research.  The  chapter  then 
summarizes  previous  research  associated  with  project  development  schedules.  The  overall 
objective  is  to  identify  key  factors  or  processes  that  influence  the  actual  time  from  decision  to 
proceed  to  delivery  of  the  first  production  item  for  military  products. 

A.  Narrowing  the  Research  Focus  from  Acquisition  Response  Time 

Narrowing  the  research  focus  to  the  process  used  to  develop  the  project  schedule  entailed 
several  intermediate  steps.  First,  the  focus  was  narrowed  to  look  at  product  development  time. 
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then  to  look  at  the  programmatic  aspects  associated  with  the  planning  of  a  project,  and  then  to  the 
process  used  to  develop  the  project  schedule.  The  rationale  for  the  focus  at  each  level  is 
discussed  below. 


A.l.  Narrowing  the  Focus  to  Development  Time 

As  discussed  in  Chapter  1,  acquisition  response  time  is  a  measure  of  the  ability  of  the 
acquisition  system  as  a  whole  to  respond  to  the  needs  of  warfighters.  Acquisition  response  time 
includes  periods-recognition,  decision,  development,  and  production-involving  significantly 
different  types  of  activities  and  groups,  as  described  in  Chapter  3.  Combining  the  different 
periods  would  obscure  important  factors  driving  acquisition  response  time,  but  addressing  all  the 
periods  would  have  been  too  broad  a  scope.  I  decided  to  focus  on  one  of  the  periods  in  detail. 
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Figure  4-1:  Components  of  Acquisition  Response  Time 

Product  development  time  appears  to  be  the  largest  component  of  acquisition  response 
time.  Development  time  starts  at  project  initiation  in  the  commercial  case  or  at  a  Milestone  1  or 
equivalent  decision  for  a  military  system,  and  ends  with  delivery  of  the  first  production  item. 
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New  major  defense  acquisition  programs  completed  between  1990  and  1994  averaged  132 
months-  1 1  years-from  project  initiation  to  first  operational  capability.67  Of  those  132  months, 
108  months  were  spent  on  the  product  development  phase,  from  the  decision  to  proceed  until 
delivery  of  the  first  production  item.  Development  time  for  major  defense  products  has 
continued  to  increase  over  the  last  20  years. 

Average  times  for  product  development  alone  are  projected  to  soon  approach  1 1  years. 
Thus  I  decided  that  for  this  research  to  effectively  reduce  the  time  required  to  field  new  military 
systems,  it  must  focus  on  product  development  time.  However,  the  time  required  for 
recognition,  decision,  and  production  is  equally  important  in  contributing  to  acquisition  response 
time.  These  areas  are  left  for  follow-on  research. 

A.2.  Narrowing  the  Research  within  Product  Development 

Many  elements  contributing  to  development  time  could  be  studied.  To  identify  which 
areas  were  likely  to  have  a  large  impact  on  development  time,  preliminary  interviews,  analysis  of 
available  data,  and  literature  reviews  were  conducted  in  both  commercial  and  military  arenas. 
This  initial  research  into  the  causes  of  long  cycle  times  led  to  a  framework  for  categorizing 
various  aspects  of  a  military  project,  and  then  to  one  area  -  the  programmatic  aspects  -  for 
further  research. 

To  provide  a  contextual  framework  for  product  development  efforts,  three  different  areas 
affecting  development  time  were  defined:  programmatic,  organizational,  and  engineering.  Each 
of  these  areas  can  be  described  at  two  levels:  the  company  level,  affecting  a  portfolio  of  projects; 
and  the  project  level,  affecting  a  particular  project. 

Programmatic  aspects  at  the  project  level  include  defining  the  project  and  the  product 
concept,  determining  available  resources;  setting  the  project  schedule,  and  establishing  the 
project’s  technical  objectives.  Other  programmatic  aspects  include  the  acquisition  strategies 


67  See  Figure  1-7 


108  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


used  to  develop  the  project,  the  acceptable  risk  associated  with  the  project,  and  the  criteria  to 
make  cost,  schedule,  and  performance»tradeoffs  within  the  project. 

The  programmatic  aspects  at  a  company  level  include  the  overall  development  strategy, 
the  strategic  planning  process  to  select  one  project,  and  the  resource  allocations  among  projects 
in  development.  Shared  among  many  of  these  aspects  are  the  number  of  projects,  the  aggregate 
funding  levels,  and  the  relationships  between  projects  in  the  portfolio.  For  defense  projects,  the 
government  (the  users,  the  Pentagon,  and  the  Program  Offices)  largely  controls  the  programmatic 
aspects  of  a  military  project  at  both  the  project  and  the  corporate  levels,  as  government  entities 
determine  the  project  objectives,  allocate  the  resources,  and  define  the  portfolio  of  development 
projects. 

Organizational  aspects  of  development  focus  on  how  the  government  and  companies 
organize  to  carry  out  the  development  projects.  At  the  project  level,  organizational  aspects  may 
include  the  use  of  integrated  product  teams,  the  seniority  of  the  program  manager,  the  level  of 
integration  of  government  and  contractor  teams,  and  the  integration  of  the  defense  contractor 
with  its  suppliers.  Decisions  regarding  who  designs  the  product  and  where  are  also 
organizational  issues.  Whether  a  particular  design  will  be  developed  in-house  or  by  suppliers  is 
one  example. 

At  the  company  or  corporate  level,  the  organizational  aspects  provide  a  framework  for 
determining  a  project’s  structure.  For  defense  projects,  the  government  often  determines  -  or  at 
least  strongly  influences  -  the  relationship  between  itself  and  the  contractors,  but  the  contractors 
and  their  suppliers  decide  how  to  organize  to  develop  the  projects. 

Engineering  aspects  include  the  methods  and  tools  used  to  design  the  project,  and  the 
companies’  manufacturing  processes.  At  the  project  level,  these  may  include  specific  design 
tools  to  be  used,  design  and  review  processes,  the  use  of  prototypes,  and  the  method  used  to 
make  cost,  schedule,  and  performance  tradeoffs.  At  the  company  level,  these  may  include  firms’ 
development  and  manufacturing  infrastructure  and  those  of  their  suppliers.  In  developing 
military  products,  the  government  has  traditionally  required  specific  tools  and  techniques,  such 
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as  a  work  breakdown  structure,  system  engineering  models,  and  information  for  periodic  review. 
The  government  has  also  determined  other  aspects  of  the  manufacturing  process  through  military 
specifications  and  standards,  but  has  recently  eliminated  many  of  these  and  moved  to 
performance-based  requirements.  While  the  Program  Offices  request  design  reviews  as  part  of 
the  oversight  of  development  contractors,  the  vast  majority  of  decisions  on  designing  and  making 
the  product  are  left  to  the  contractors  and  their  suppliers. 

Review  of  a  large  number  of  papers  and  books  on  commercial  product  development 
revealed  significant  effort  devoted  to  improving  all  three  areas  -  programmatic,  organizational, 
and  engineering.  Figure  4-2  shows  how  recent  commercial  efforts  to  improve  product 
development  map  against  this  framework,  at  the  company  and  the  project  level. 
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How  to  defined  a  project  and  How  to  organize  to  develop  the 
allocate  resources  project 


How  to  develop  the  project  design  and 
manufacturing  processes 


Company  Planning  Level  Company  Organization  Level 


Company  Engineering  Level 
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Figure  4-2:  Mapping  Efforts  to  Improve  Product  Development  onto  the  Programmatic, 
Organizational,  and  Engineering  Categories 


To  significantly  reduce  development  time,  one  would  expect  changes  would  be  needed  in 
each  area.  However,  some  areas  do  not  currently  limit  development  time.  As  the  development 
process  improves,  the  limiting  constraint  will  likely  change.  Since  the  overall  goal  is  a 
significant  cut  in  development  time,  items  in  each  area  will  likely  need  to  be  addressed  to  make  a 
major  dent. 
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A.3  Relationship  to  Other  Lean  Aerospace  Initiative  Research  - 

This  thesis  is  part  of  the  larger  Lean  Aerospace  Initiative  (LAI),  which  encompasses 
many  areas  of  product  development.  LAI  research  has  so  far  focused  primarily  on  contractor 
organizations  and  what  they  can  do  to  become  lean.  This  research  has  concentrated  on 
organizational  and  engineering  aspects  of  development  time,  not  on  government-related 
activities.  Figure  4-3  shows  the  mapping  of  LAI  research  against  the  proposed  framework. 
Because  each  development  project  may  be  limited  by  different  factors,  research  in  all  three  areas 
is  required. 
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Figure  4-3:  Mapping  Research  under  the  Lean  Aerospace  Initiative  onto  the 
Programmatic,  Organizational,  and  Engineering  Categories 


In  the  organizational  and  engineering  areas,  there  is  significant  overlap  between  military 
and  commercial  development  projects.  Some  of  this  overlap  can  be  seen  in  the  focus  over  the 
past  decade  on  using  integrated  product  teams  and  advanced  engineering  tools  and  practices  in 
both  commercial  and  defense  systems.  Yet  only  commercial  industries  have  significantly 
reduced  development  times,  while  those  for  defense  systems  have  continued  to  increase. 
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Long  development  times  are  evident  across  all  types  of  military  projects,  including 
aircraft,  electronics,  munitions,  spacecraft,  radar,  and  helicopters.  Each  of  these  areas  is  run  by 
different  Program  Offices  and  defense  contractors,  all  of  which  use  somewhat  contrasting 
approaches  to  organizational  and  engineering  aspects.  Significant  effort  has  also  already  been 
applied  in  the  engineering  and  organizational  areas.  This  indicates  that  something  more 
systematic  may  be  affecting  development  times,  and  that  the  organizational  and  engineering 
areas  may  not  be  the  limiting  aspects. 

Based  on  initial  observations,  it  appears  that  further  improvements  in  organizational  and 
engineering  areas,  without  changes  in  the  programmatic  area,  are  unlikely  to  yield  desired 
reductions  in  time  to  develop  new  military  products.  Yet  little  research  has  been  conducted  in 
this  area.  I  therefore  decided  to  examine  programmatic  aspects  to  determine  if  they  could  be  the 
key  to  reducing  development  times  for  military  systems. 


B.  Programmatic  Aspects  and  Development  of  the  Project  Schedule 

The  programmatic  area  of  a  project  includes  many  potential  areas  for  in-depth  research. 
They  include  the  methods  and  processes  used  to  determine  a  project’s  objectives,  allocation  of 
resources  to  a  project,  selection  of  which  projects  to  initiate,  and  determination  of  a  project’s 
schedule.  The  initial  investigation  revealed  that  many  of  the  programmatic  aspects  of  a  project 
appear  to  be  determined  very  early  in  a  project’s  planning  phase.  Numerous  discussions  with 
government  and  contractor  participants  in  the  LAI  revealed  that  one  area  -  the  process  used  to 
determine  a  project’s  schedule  -  appears  most  directly  related  to  time  to  develop  a  product. 

Initial  investigation  also  indicated  that  an  in-depth  analysis  of  that  process  would  yield 
significant  insight  into  the  factors  driving  development  times.  A  focus  on  a  project’s  initial 
schedule  can  reveal  the  processes,  influences,  information,  and  organizations  involved.  It  also 
appeared  that  an  analysis  of  the  process  to  develop  and  execute  a  project’s  schedule  would  clarify 
the  impact  of  other  programmatic,  organizational,  and  engineering  aspects  of  development  time, 
and  identify  the  primary  causes  of  long  times.  Thus  I  decided  to  focus  this  research  on  the 
process  used  to  develop  a  project’s  schedule,  and  its  impact  on  eventual  development  time. 
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B.l.  Preliminary  Research  Observations  on  Schedule  Development  Issues 

In  trying  to  identify  the  causes  of  long  cycle  times  and  narrow  the  research,  I  interviewed 
many  people,  reviewed  relevant  literature,  and  uncovered  sources  of  schedule-related  data. 

The  potential  causes  of  long  development  time  identified  in  initial  interviews  and 
research  included  advanced  technology,  aggressive  requirements,  complex  users’  needs,  funding 
limitations,  contractors’  capabilities,  testing,  support  requirements,  and  training  requirements. 
People  interviewed  at  the  Pentagon,  Program  Offices,  and  defense  contractors  held  disparate 
views  of  the  causes  of  long  development  time  and  their  impact.  There  was  also  a  wide  range  of 
views  on  the  importance  of  shortening  project  schedules,  ranging  from  those  who  considered  that 
effort  critical  to  those  who  thought  it  irrelevant.  Those  interviewed  had  little  information  on  the 
development  of  project  schedules  other  than  for  those  in  which  they  were  directly  involved. 

The  primary  focus  of  existing  studies  and  those  interviewed  appeared  to  be  meeting 
schedule  and  minimizing  slip.  Reducing  schedule  beyond  the  existing  plan  was  not  a  significant 
priority  and,  in  many  cases,  not  considered  an  objective  at  all.  Little  research  was  available  on 
the  development  of  the  initial  project  schedule. 

Interviews  with  industry  and  government  representatives  revealed  that  the  Program 
Office’s  plans  and  expectations  appeared  to  significantly  affect  a  project’s  initial  schedule 
Industry  representatives  indicated  that  the  Program  Office’s  expected  schedule  played  a  large 
role  in  their  proposed  project  schedules.  One  industry  representative  said  that  the  government 
had  to  tell  him  what  type  of  program  to  bid  and  provide  guidelines  to  help  him  be  responsive  to 
the  government’s  desires.  Many  industry  representatives  indicated  they  had  little  incentive  to  bid 
anything  other  than  the  expected  schedule.  Several  indicated  they  had  a  disincentive  to  do  so, 
since  they  feared  they  would  be  seen  as  unresponsive  and  assumed  to  pose  a  higher  risk,  and 
would  thus  be  less  likely  to  win  the  contract.  Winning  the  development  contract  was  companies’ 
central  goal.  When  they  do  not  win  the  contract,  they  are  essentially  locked  out  of  the  market  for 
that  product  forever. 


1 14  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Many  people  stated  that  few  positive  incentives  related  to  project  schedules  exist.  No 
documentation  addressing  incentives  related  to  project  schedules  could  be  found.  The  schedule- 
related  data  that  were  available  from  a  number  of  sources  indicated  that  many  projects  were 
completed  on  schedule  and  few  were  ever  completed  early. 

Initial  research  indicated  that  schedule  development  was  likely  to  exert  a  strong  impact 
on  the  project’s  outcome  but  little  supporting  data  were  available.  There  was  no  agreement  on 
the  factors  involved  in  determining  a  project’s  initial  schedule,  or  on  the  structure  of  the  process 
involved.  Research  into  the  process  used  to  develop  the  initial  schedule  appeared  to  have 
significant  potential  to  answer  continuing  questions  and  identify  which  factors  drive  project 
development  times. 

B.2.  Initial  Research  Questions 

Questions  based  on  the  initial  research  observations  were  developed  to  establish  the  scope 
and  direction  of  the  research.  One  set  of  questions  dealt  with  the  development  of  the  initial 
project  schedule.  How  are  these  initial  schedules  determined?  Who  is  involved?  What  schedule 
information  and  tools  are  used?  And  what  factors  are  considered  important? 

A  second  line  of  questions  dealt  with  how  these  initial  project  schedules  affect 
development  of  the  contractor’s  schedules  and  the  eventual  contracted  schedule.  What  impact 
does  the  project’s  initial  schedule  have  on  the  contractor’s  proposed  schedule?  What  are  the 
incentives  associated  with  the  contractor’s  proposed  schedule?  What  are  the  results  of  this 
process  when  comparing  the  government’s  initial  schedule  with  the  eventual  contracted 
schedules? 

A  third  line  of  questions  dealt  with  the  impact  of  the  initial  and  contract  schedules  on  a 
project’s  actual  development  schedule.  How  do  a  project’s  initial  schedule  and  contract  schedule 
affect  the  actual  development  schedule?  What  are  the  schedule-related  incentives  during  the 
development  phase?  Are  there  significant  barriers  to  shortening  project  schedules  and  actual 
development  times? 
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These  represent  only  a  few  of  the  questions  that  must  be  answered  to  understand  the 
process  used  to  develop  the  project  schedule,  and  the  impact  of  the  initial  schedules  on  actual 
development  time. 

B.3.  The  Relationship  between  Project  Schedules  and  Development  Time 

It  is  important  to  clearly  define  the  relationship  between  project  schedules  and  product 
development  times.  Project  schedules  set  the  major  milestones:  they  are  forward  projections  of 
activities  that  will  be  accomplished  as  part  of  the  development  effort.  Project  schedules  are  not 
an  objective  measure  but  are  intended  to  estimate  accurately  the  time  it  will  take  to  develop  a 
product,  based  on  the  perceived  and  observed  constraints  placed  on  the  Program  Office  and  the 
contractors. 

For  this  research  a  project’s  initial  schedule  is  defined  as  the  schedule  used  for  a 
Milestone  1  decision,  or  for  approval  of  project  initiation.  This  schedule  is  used  to  plan  the  many 
aspects  required  to  develop  and  field  military  products.  A  project’s  initial  schedule  is  developed 
during  early  acquisition  planning  activities  in  the  Pre-Milestone  0  or  phase  0  period,  and  are 
included  in  the  acquisition  Milestone  1  decision  or  its  equivalent.  The  project  schedule  becomes 
the  basis  for  planning  activities  at  many  levels,  including  those  of  the  contractors,  the  testers,  the 
funding  organizations,  and  the  users.  Other  significant  schedules  include  the  contract  schedule 
between  the  government  and  the  contractor  to  develop  the  product,  and  the  actual  development 
schedule  as  the  project  moves  forward. 

A  project’s  initial  schedule  is  not  necessarily  an  estimate  of  the  minimum  time  required 
to  develop  the  project,  assuming  that  all  unnecessary  activities,  constraints,  and  delays  have  been 
eliminated.  The  minimum  required  time  would  be  based  primarily  on  the  time  necessary  to 
define  the  requirements,  develop  the  required  technology,  design  the  system  to  meet  the 
requirements,  and  develop  the  process  to  manufacture  the  system.  Unfortunately,  it  is  difficult  to 
identify  the  period  actually  required  to  develop  a  product  without  a  good  example  ahead  of  time 
or  a  retrospective  analysis. 
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The  initial  schedule  is  not  just  an  unbiased  estimate  of  the  time  required  to  develop  a 
project-it  appears  to  affect  the  time  it  takes  to  develop  the  project.  If  initial  schedule  were 
simply  an  unbiased  estimate,  one  would  expect  a  distribution  of  projects  delivered  before  and 
after  the  date  determined  by  the  initial  schedule.  But  Figure  1-8,  which  compares  planned 
schedules  with  eventual  development  times,  shows  that  50  percent  of  projects  meet  their  initial 
schedule,  many  are  completed  behind  schedule,  and  very  few  finish  early.  The  fact  that  so  many 
projects  finish  exactly  on  their  initial  schedule  appears  to  indicate  either  that  those  who  develop 
initial  schedules  are  very  good  at  estimating  the  minimum  time  required,  or  that  a  project’s  initial 
schedule  affects  the  minimum  time  it  actually  takes.  The  two  are  not  necessarily  exclusive 
explanations.  The  extent  and  the  mechanisms  of  this  impact  are  one  of  the  key  areas  of  this 
study. 


If  one  were  to  develop  the  most  accurate  initial  schedule,  the  processes  used  and  the 
factors  considered  in  developing  it  would  be  very  important.  If  one  were  interested  in 
developing  a  product  as  quickly  as  effectively  and  efficiently  as  possible,  one  would  expect 
technology  development,  engineering  requirements,  and  manufacturing  development  to  play  a 
central  role  in  determining  project  schedules.  Reducing  the  time  required  for  those  areas  has 
been  a  major  focus  of  many  commercial  firms  and  academic  researchers.  Most  military 
initiatives  have  focused  on  integrated  product  and  process  development,  integrated  product 
teams,  concurrent  engineering,  rapid  prototyping,  prototype  production  lines,  key  characteristics, 
computer-aided  design,  and  computer-aided  manufacturing.  All  these  initiatives  are  aimed  at 
reducing  the  engineering  and  organizational  time  required  to  develop  products.  But  if  those 
areas  are  not  central  to  the  schedule  development  process,  they  probably  will  not  have  the 
expected  impact  on  actual  development  time. 


C.  Relation  to  Previous  Schedule-Related  Research  on  Defense  Projects 

Many  people  at  distinguished  organizations  have  examined  development  times  for 
defense  projects.  But  none  have  addressed  the  development  of  the  project’s  initial  schedule,  or 
its  impact  on  development  time  it.  Most  researchers  have  investigated  the  ability  of  project 
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managers  to  meet  planned  schedules  and  reduce  delays.  Several  studies  have  examined  the 
effects  of  acquisition  reform  on  reducing  cycle  time  and  improving  schedule  performance.  Most 
studies  have  relied  on  case  studies  although  some  have  used  statistical  analysis  of  databases. 
These  reports  focus  on  the  duration  of  projects,  the  deviation  from  the  expected  plan,  and  the 
estimating  techniques  for  several  types  of  weapon  systems. 

C.1  Research  on  Schedule  Duration  and  Slip 

It  is  difficult  to  separate  the  studies  that  examined  schedule  duration  and  slip,  as  the  same 
report  often  includes  both  topics.  Below  is  a  quick  summary  of  the  research  in  the  area. 

AFSC  Affordable  Acquisition  Approach  Study 

The  Affordable  Acquisition  Approach  Study,  completed  in  1983  by  the  Air  Force 
Systems  Command  (AFSC),  began  as  the  Accelerated  Acquisition  Approach  Study.  This  study 
focused  on  whether  projects  actually  take  longer  than  scheduled,  or  whether  that  is  simply  the 
perception.  The  study  also  investigated  the  reasons  for  any  delays,  and  techniques  for  shortening 
the  process  of  procuring  systems  at  lower  cost.  The  research  was  based  on  literature  reviews, 
interviews,  and  analysis  of  109  acquisition  projects.  The  study  found  that  development  and 
production  times  had  increased  significantly  over  30  years.  The  study  identified  the  major  cause 
as  funding  instability  caused  by  budget  cuts  and  ballooning  project  cost.  The  study  called  for 
longer-term  planning,  more  program  stability,  and  fewer  projects.  The  study  also  recommended, 
as  well  as  the  use  of  program  baselines,  firm  commitments  at  acquisition  milestones,  and 
program  management  tools.  The  study  did  not  address  development  of  project  schedules  but 
only  their  execution.  Of  note,  the  manager  for  this  effort  was  Dr.  Jacques  Gansler,  now 
Undersecretary  of  Defense  for  Acquisition  and  Technology. 

Rand  Research 

The  Rand  Corp.  has  pursued  several  research  efforts  examine  acquisition  costs  and 
schedules.  In  a  1980  study  of  67  programs,  Giles  Smith  and  Friedmann  found  that  the 
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demonstration  validation  phase  was  growing  by  10  months  per  decade.  They  found  that  the 
time  required  for  full-scale  development  had  not  significantly  changed,  but  that  the  production 
phase  had  increased  by  6  months  per  decade.69  This  was  a  statistical  analysis  of  programs  based 
on  schedule  outcomes.  The  authors  did  not  evaluate  how  the  schedules  were  developed.  In  a 
Rand  follow-up  study  in  1987,  Rothman  updated  the  database  and  looked  for  effects  of  the 
starting  date  on  full-scale  development  but  did  not  find  a  significant  correlation.70 

In  1990,  Rand  issued  a  report  by  Jeff  Drezner  and  Giles  Smith  entitled  An  Analysis  of 
Weapon  Acquisition  Schedules.  This  report  used  a  case  study  method  to  analyze  the  major 
drivers  of  schedules  and  deviations  from  the  plan  for  10  major  defense  acquisition  programs. 

The  authors  looked  primarily  at  programmatic  factors  such  as  competition,  prototyping,  military 
priority,  funding  adequacy,  and  joint  service  management.  However,  they  found  that  the  lack  of 
documentation  on  the  rationale  for  the  original  plan  limited  their  insight  into  the  factors  affecting 
that  plan  and  why  the  actual  program  deviates  from  the  plan.  They  also  found  that  because  of  the 
small  number  of  programs  studied,  meaningful  statistical  analysis  on  many  variables  was  not 
possible.  Their  approach  was  therefore  generally  heuristic.71 

Drezner  and  Smith  identified  four  factors  with  discernible  but  not  measurable  effect  on 
the  original  schedule.  They  found  that  competition  in  contracting  generally  lengthens  schedules, 
concurrency  or  overlapping  between  development  and  production  shortens  schedules,  use  of 
prototypes  lengthens  schedules,  and  making  a  project  a  service  priority  shortens  schedules.  The 
authors  could  not  discern  effects  from  funding  adequacy,  separate  contracting  actions  between 
development  phases,  external  guidance,  joint  program  management,  program  administrative 
complexity,  technical  difficulty,  and  stability  of  system  requirements.72  The  authors  also  found 


68  Smith,  Giles  and  Friedman,  E.T.  “An  Analysis  of  Weapon  Systems  Acquisition  Intervals,  Past  and  Present.” 
Contract  F49620-77-C-0023,  MDA903-78-C-0188  Santa  Monica:  The  RAND  Corporation,  November  1980  (R- 
2605-DR&E/AF).  Pg  15  and  25. 

69  Smith  and  Friedman  1980  Pg  30. 

70  Rothman,  M.B.  “Aerospace  Weapons  System  Acquisition  Milestone:  A  Data  Base  .  Contract  MDA903-85-C- 
0030.  Santa  Monica  CA:  The  RAND  Corporation,  October  1987  (N-2599-ACQ)  Pg.  15. 

71  Drezner,  J.  and  Smith  G.  “An  Analysis  of  Weapon  System  Acquisition  Schedules”.  R-3937-ACQ.  Santa  Monica 
CA:  The  RAND  Corporation.  Dec  1990.  (R-3937-ACQ) 

72  Drezner  and  Smith  Pg.  30. 
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the  lack  of  data  on  schedule-related  issues  a  significant  barrier  to  research  on  program  schedules 
and  the  causes  of  slip.  Because  documentation  was  lacking,  they  reported  they  could  not 
evaluate  the  adequacy  of  the  projects’  initial  schedules. 

In  1996,  Rand  released  a  significant  database  documenting  the  schedules  and  cost 
performance  of  several  hundred  major  defense  acquisition  projects.73  Although  tthe  study  limited 
schedule  variables  to  major  milestones  and  decision,  this  database  provides  the  most 
authoritative  measure  of  schedule  performance  compared  with  planned  schedule.  Unfortunately, 
it  does  not  provide  insight  into  the  factors  involved  in  developing  a  project’s  initial  schedule. 

Institute  for  Defense  Analysis 

In  1989  Karen  Tyson,  Richard  Neslon,  Neang  Om,  and  Paul  Palmer  issued  a  report 
examining  the  outcome  of  cost  and  schedule  plans  and  the  effects  of  acquisition  reform  activities 
on  cost  and  schedule  growth.74  These  authors  evaluated  the  effects  of  six  factors— prototyping, 
competition,  multi-year  procurement,  design  to  cost,  sole-source  procurement,  fixed-price 
development,  and  contract  incentives— on  89  major  acquisition  programs  They  did  not  evaluate 
the  adequacy  or  the  development  of  the  project  schedules. 

The  Analytical  Sciences  Corp. 

In  1986  and  1987  The  Analytical  Science  Corp.  released  two  papers  on  possible  schedule 
drivers.75  76  These  reports  identified  six  potential  schedule  drivers:  technical  complexity,  degree 
of  technological  change,  system  mission,  procurement  period,  acquisition  strategy,  and  funding 
profile.  The  first  report  found  the  physical  parameters  of  a  system  marginally  significant  in 


75  Jarvaise,  Drezner,  and  Norton.  The  Defense  System  Cost  Performance  Database:  Cost  Growth  Analysis  Using 
Selected  Acquisition  Reports.”  MR-625-OSD  Santa  Monica  CA:  The  RAND  Corporation.  1996. 

74  Tyson,  Karen,  Richard  Nelson,  Neang  OM,  and  Paul  Palmer.  Acquiring  Major  Systems:  Cost  and  Schedule 
Trends  and  Acquisition  Initiative  Effectiveness.  Contract  MDA903-84-C-0031.  Alexandria  VA:  Institute  for 
Defense  Analysis,  March  1989  (P-2201) 

75  Nelson,  Eric  K.  Independent  Schedule  Assessment  FSD  Study”  Contract  F33657-82-D-0064  Fairborn  OH  The 
Analytical  Sciences  Corporation.  30  June  1986.  (TR-5300-2-2) 

76  Nelson,  Eric  and  Trageser,  Jay.  “Analogy  Selection  Methodology  Study”.  The  Analytical  Sciences  Corporation. 
Fairborn  OH.  Contract  F33657-82-D-0064  1  December  1987  (TR-5306-7-2) 
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affecting  schedule.  Neither  effort  was  able  to  identify  a  combination  of  parameters  that  could 
account  for  the  large  variance  in  the  Engineering  and  Manufacturing  Development  schedules.77 

Air  Force  Institute  of  Technology 

The  research  most  closely  related  this  thesis  was  conducted  by  two  British  Air  Force 
officers,  Richard  Hazeldean  and  John  Topfer,  who  did  a  master’s  thesis  on  the  effect  of 
contracting  actions  on  eventual  schedule  performance  of  smaller  development  programs.78  These 
authors  painstakingly  gathered  schedule  data  on  25  smaller  projects  from  contract  records, 
finding  that  smaller  contracts  posted  worse  schedule  performance  than  larger  contracts.  The 
authors  also  found  that  several  pre-contract  factors  affected  schedule  performance:  advance 
scheduling,  the  level  of  concurrency  between  development  and  production,  the  preliminary  work 
breakdown  structure,  use  of  a  network  format  to  present  schedule  information,  and  the  number  of 
reports  required.  The  authors  also  found  that  the  methods  used  to  manage  the  schedule  were  not 
well  understood  by  Program  Offices.  They  cited  It  many  observations  from  their  efforts  to 
collect  data:  Considerable  schedule  information  was  buried  within  a  number  of  different  RFP  and 
proposal  sections. 

•  Schedule  management  requirements  were  rarely  specified  in  a  coherent  and 
integrated  manner. 

•  The  work  breakdown  structure-a  primary  schedule  document-was  usually  hidden  in 
the  cost  section  of  the  proposal,  implying  a  primary  cost  usage  as  opposed  to 
schedule  focus. 

•  Schedule  management  requirements  were  rarely  proactive  but  were  reactive  to 
problems  once  they  occurred. 

•  Only  a  limited  number  of  techniques  for  managing  the  schedule  were  specified,  so 
sophisticated  scheduling  tools  such  as  PERT  or  contingency  schedules  were  rarely 
used. 

•  The  schedule  management  sections  in  the  RFP  and  contracts  appeared  to  be  derived 
from  the  “Copy-From”  principle  taken  word-for-word  from  previous  documents. 


77  Nelson,  Eric  and  Trageser,  Jay.  “Analogy  Selection  Methodology  Study”.  The  Analytical  Sciences  Corporation. 
Fairborn  OH.  Contract  F33657-82-D-0064  1  December  1987  (TR-5306-7-2)  Page  5-1. 

78  Hazeldean  Richard,  and  Topfer  John,  “Contracting  for  Schedule  Performance:  The  Relationship  Between  Pre¬ 
contract  Award  Actions  by  the  DoD  and  the  Resultant  Schedule  Performance.  AFIT  Thesis  Wright  Patterson  AFB 
OH.  1993. 
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The  authors’  efforts  were  hampered  by  the  lack  of  schedule-related  data  for  many  programs 
and  the  difficulty  of  gathering  data  that  did  exist.  Hazeldean  and  Topfer  did  not  address  issues 
related  to  developing  the  initial  schedule,  concentrating  instead  on  the  effect  of  contracting- 
related  actions  on  adherence  to  the  schedule. 

C.2  Research  on  Schedule-Estimating  Relationships 

Bruce  Harmon,  Lisa  Ward,  and  Paul  Palmer  of  the  Institute  for  Defense  Analysis  have 
attempted  to  estimate  the  time  required  to  develop  new  aircraft  and  missile  systems.79  80  The 
system  parameters  they  studied  included  an  item’s  weight,  the  number  of  engines,  the  wing  area, 
the  percentage  of  titanium,  and  many  other  variables.  Programmatic  variables  included 
prototyping,  contractor  teaming,  service,  and  contractor.  These  authors  are  now  attempting  to 
uncover  a  simple  relationship  between  a  system’s  physical  characteristics  and  its  schedule,  much 
like  the  apparent  correlation  between  weight  and  cost  of  an  aircraft.  But  no  such  relationship  has 
so  far  been  found.  The  authors  have  found  that  programmatic  aspects  play  a  significantly  larger 
role  than  system  characteristics  in  determining  project  schedules  and  development  times.  Major 
drivers  include  funding  levels,  the  use  of  prototypes,  and  the  use  of  contractor  teaming.  Analysis 
of  statistical  relationships  is  limited  by  the  lack  of  available  data  and  the  small  number  of 
projects  studied,  which  included  9  aircraft  and  14  missile  programs. 

Scott  Boyd  and  Brian  Mundt  found  similar  results  when  they  focused  on  56  efforts  to 
develop  transport,  tanker,  and  bomber  aircraft.  The  authors  found  that  only  two  variables  they 
studied  affected  development  schedule:  whether  the  project  entailed  modifying  rather  than 
creating  an  aircraft;  and  whether  the  project  included  a  prototype  phase.  No  physical 
characteristic  of  the  system  affected  time  from  the  start  of  Engineering  and  Manufacturing 


79  Harmon,  Bruce,  Lisa  Ward,  and  Paul  Palmer.  “Assessing  Acquisition  Schedules  for  Tactical  Aircraft,”  Cost 
Analysis  Applications  of  Economics  and  Operations  Research  (proceedings  of  the  Institute  of  Cost  Analysis 
National  Conference),  259-280.  New  York  NY:  Springer-Verlag  1989. 

80  Harmon,  Bruce  and  Lisa  Ward.  “Schedule  Estimating  Relationships  for  Air-Launched  Missiles,”  Cost  Analysis 
and  Estimating:  Tools  and  Techniques  (Proceedings  of  the  Institute  of  Cost  Analysis  National  Conference),  1  IS¬ 
IS?.  New  York  NY:  Springer-Verlag.  1990. 
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Development  (EMD)  to  delivery  of  the  first  production  item.81  The  lack  of  available  data  from 
which  to  base  the  analysis  was  cited  as  the  most  significant  impediment  to  developing  effective 
schedule  estimating  relationships. 

C.3  Other  Relevant  Studies 
The  Packard  Commission82 

A  central  focus  of  the  Packard  Commission  was  comparing  acquisition  times  between 
similar  commercial  and  the  military  projects.  The  commission  found  that  military  development 
efforts  took  considerably  longer  and  cost  significantly  more.  The  commission  concluded  that 
long  cycle  times  were  a  central  problem  in  the  acquisition  process  and  caused  many  other 
problems.  The  commission  also  concluded  that  military  development  times  were  excessively 
long  and  could  be  cut  in  half.  It  made  specific  recommendations  for  changing  the  structure  of 
acquisition  organizations,  which  were  quickly  implemented.  But  despite  these  changes, 
acquisition  schedules  have  not  been  reduced.  Interviews  reveal  that  the  commission’s  schedule- 
related  goals  are  not  widely  recognized. 

Defense  Science  Board  Reports 

Many  Defense  Science  Board  reports  have  addressed  development  times,  beginning  with 
the  1977  report  of  the  Acquisition  Cycle  Time  Task  Force.  Many  reports  have  since  focused  on 
acquisition  reform  efforts,  the  use  of  commercial  practices  and  components,  performance-based 
specifications,  and  research  and  development  strategies.  Other  studies  have  looked  at 
development  and  manufacturing  tools  and  practices.  Analysis  has  been  largely  based  on  case 
studies,  briefings  on  various  efforts,  and  the  judgment  of  the  people  involved.  None  of  the 
Defense  Science  Board  reports  have  looked  at  the  schedule  development  process  and  its  impacts 
on  development  time. 


“  Scott  Boyd  and  Brian  Mundt  Schedule  estimating  relationships  for  the  Engineerng  and  Manufacturing 
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C. 4  Conclusions  of  the  Literature  Review 

Many  respected  groups  have  made  significant  efforts  to  identify  causes  of  long 
development  times.  Most  studies  have  focused  on  the  causes  of  deviations  from  planned 
schedules  or  on  total  development  time.  A  few  have  tried  to  estimate  schedules  based  on  project 
characteristics.  Most  studies  report  being  hampered  by  a  lack  of  detailed  schedule-related  data. 
No  studies  have  addressed  the  process  used  to  develop  a  project’s  initial  schedule  or  its  impact 
on  development  time. 

D.  Schedule  Development  Overview 

D.l  Phases  of  Schedule  Development 

The  three  main  phases  entailed  in  developing  and  executing  a  project  schedule  include 
the  planning  phase,  the  contracting  phase,  and  the  development  phase.  Activities  in  each  phase 
affect  the  downstream  phases. 

Planning  Phase 

The  planning  phase  occurs  in  a  project’s  initial  stages,  when  rough  schedules  are 
developed  to  help  organize  the  eventual  effort.  During  this  phase  acquisition  plans,  strategies, 
and  schedules,  funding  profiles,  and  program  objectives  are  set.  Development  of  the  initial 
project  schedule  establishes  the  initial  program  objectives.  These  activities  usually  occur  before 
the  Milestone  I  decision  to  initiate  the  project. 

Contracting  Phase 

During  this  phase  the  development  contractor  is  selected  and  awarded  a  contract  based  on  the 
initial  plans.  This  phase  includes  developing  the  requirements  and  objectives  to  be  part  of  the 
contract,  developing  the  RFP,  developing  the  contractor  proposals,  and  selecting  an  appropriate 
contractor  and  negotiating  the  contract.  Through  this  process,  the  initial  project  schedule  is 
converted  to  a  contract  schedule  that  is  binding  on  the  government  and  the  contractors.  The 
contract  schedule  can  be  changed  only  by  amending  the  contract. 
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Development  Phase 

Once  the  contract  is  signed,  the  development  phase  begins,  and  lasts  until  delivery  of  the  first 
product.  This  period  may  include  developing,  testing,  and  producing  prototypes,  and  other 
actions  that  affect  the  design  of  the  product  and  the  process  used  to  make  it.  The  schedule  for  this 
period  is  largely  defined  within  the  integrated  master  schedule  and  the  work  breakdown  structure 
incorporated  into  the  contract.  Adherence  to  the  schedule  is  tracked  through  a  variety  of 
methods,  including  technical  and  financial  reviews. 

These  three  phases  are  followed  by  a  production  phase  that  yields  the  required  number  of 
items.  Schedules  in  the  production  phase  affect  the  time  required  to  reach  operational  capability, 
but  they  are  not  a  focus  of  this  study.  For  more  information  on  production  cycle  times,  see  other 
research  conducted  under  the  purview  of  the  Lean  Aerospace  Initiative. 
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Figure  4-4:  Schedule  Development  Phases 


D.2.  Actors  and  Organizations  Involved  in  Developing  and  Executing  Project  Schedules 
As  described  in  Chapter  3,  the  product  development  process  includes  four  primary 
organizations:  the  users,  the  Pentagon,  the  Program  Office,  and  the  defense  contractor.  These 
same  organizations  play  a  critical  role  in  developing  the  schedule  and  determining  the  duration  of 


83  Lean  Aerospace  Initiative  research  is  available  through  the  LAI  web  site  at  http://lean.mit.edu 
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the  development  effort.  These  four  organizations  also  contribute  to  defining  and  executing  the 
development  effort. 

Users 

The  users  are  typically  the  major  military  command  that  will  employ  the  equipment.  The  users 
develop  modernization  plans  and  requirements,  as  well  as  and  a  preliminary  program  objective 
memorandum  outlining  the  preferred  allocation  of  resources.  The  individuals  responsible  for  a 
given  project  are  spread  over  several  organizations:  major  command  headquarters  typically 
includes  a  planning  organization,  a  requirements  organization,  and  a  funding  organization. . 

i 

The  Pentagon 

The  Pentagon  organizations  key  to  developing  program  schedules  are  those  involved  in  the 
requirements,  funding,  and  acquisition  processes.  The  primary  interfaces  between  individual 
projects  and  other  various  groups  within  the  Pentagon  are  the  program  element  monitor,  the 
acquisition  action  officer,  and  the  requirement  action  officer.  These  officers  are  spread  between 
offices  of  the  assistant  secretary  of  the  Air  Force  for  acquisition,  and  the  Air  Force  deputy  chief 
of  staff  for  operations,  depending  on  the  operational  status  of  the  weapon  system..  The  various 
offices  in  the  Pentagon  make  the  final  decisions  on  requirements,  funding  levels,  and  major 
acquisition  issues 

The  Program  Offices 

The  Program  Offices  are  the  primary  acquisition  organizations  in  the  Air  Force,  and  are 
responsible  for  contracting  for  product  development.  Each  Program  Office  is  organized  around  a 
major  weapon  system  or  set  of  similar  systems.  The  Program  Office  is  led  by  a  program 
director,  typically  a  colonel.  The  average  Program  Office  has  many  development  programs 
under  way,  ranging  from  a  major  aircraft  project  to  small  upgrades.  Each  effort  is  led  by  a 
program  or  project  manager,  whose  rank  depends  on  the  project’s  size,  scope,  and  cost;  and  an 
associated  team.  The  Program  Office  and  the  project  manager  are  the  primary  links  to  the 
contractor,  and  they  develop  the  acquisition  plans,  the  RFP,  and  source  selection  criteria. 
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Defense  Contractor 

The  early  stages  of  a  project  may  include  a  number  of  competing  development  contractors.  The 
contractor’s  proposal  includes  a  detailed  schedule  of  the  actions  to  be  undertaken  as  part  of  the 
contract.  This  schedule  becomes  the  basis  for  the  contracted  schedule  used  throughout  the 
development  phase.  The  central  individual  within  the  contractor  organization  is  the  program 
manager — typically  the  proposal  manager,  who  has  been  very  involved  in  establishing  the 
proposed  schedule.  Once  on  contract,  the  program  manager  must  ensure  that  the  cost,  schedule, 
performance,  and  financial  objectives  are  met. 

Other  organizations  that  play  a  role  in  the  schedule  are  the  Department  of  Defense, 
suppliers,  testing  organizations,  personnel  organizations,  training  organizations,  and  logistics 
organizations.  These  organizations  play  a  secondary  role  in  determining  schedules. 


Chapter  Summary 

No  significant  effort  has  attempted  to  characterize  the  process  used  to  develop  the  initial 
schedule  for  military  development  projects.  Preliminary  research  conducted  as  part  of  this  thesis 
indicated  that  the  initial  project  schedule  may  play  a  significant  role  in  determining  the  time 
required  to  develop  projects.  Others  within  the  Lean  Aerospace  Initiative  are  exploring 
organizational  and  engineering  avenues  for  reducing  product  development  time.  This  indicates 
that  research  on  development  of  the  initial  project  schedule  and  its  impact  an  appropriate  focus  of 
considerable  effort. 


Chapter  5 


Research  Methodology _ 

A.  Exploratory  versus  Hypothesis-Driven  Research 

The  two  major  approaches  to  studies  of  this  type  are  exploratory-based  and  hypothesis-based 
research.  Exploratory  research  is  a  broader,  more  open-ended  approach  that  allows  examination 
of  many  factors  that  may  affect  the  process  in  question.  Hypothesis-driven  research  establishes  a 
presumed  mode  by  which  a  system  operates  and  then  proves  or  disproves  the  hypothesis. 
Hypothesis-driven  research  is  very  effective  when  building  on  research  in  areas  with  several 
theories  of  how  a  system  works.  In  the  area  of  developing  initial  project  schedules,  there  is  no 
apparent  consensus  and  little  previous  research  on  how  the  process  works.  Any  attempts  to 
define  a  testable  hypothesis  before  doing  the  research  would  not  seem  to  stand  a  high  chance  of 
success  in  addressing  all  the  issues  that  may  affect  initial  project  schedules.  There  are  also 
widely  varying  opinions  of  causes  of  long  cycle  times  for  defense  systems.  Exploratory  research 
allows  many  more  leads  and  avenues  of  investigation  to  be  followed  and  analyzed.  Because  of 
these  factors,  I  decided  that  my  research  into  the  schedule  development  process  and  its  impacts 
would  have  a  broad  scope.  It  would  attempt  to  explain  how  the  schedule  development  processes 
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works,  and  characterize  the  impacts  on  development  time,  rather  than  define  a  few  specific 
hypotheses  that  might  or  might  not  prove  relevant  when  the  data  are  analyzed. 

B.  Level  or  Unit  of  Analysis 

The  individual  development  project  was  chosen  as  the  appropriate  level  of  analysis  for 
this  research.  The  typical  approach  has  been  to  regard  schedules  and  schedule  slip  for  major 
acquisition  programs  as  a  single  entity.  This  typically  has  limited  the  analysis  to  major  defense 
acquisition  programs  or  weapon  systems.  But  the  program-level  data  can  hide  the  factors  driving 
the  schedules  of  various  subordinate  development  projects.  Those  data  may  also  mask  specific 
problems  and  issues  encountered  with  one  project  within  an  overall  program. 

Analysis  of  individual  development  projects  allows  a  larger  number  of  projects  to  be  studied,  and 
the  factors  driving  each  project  to  be  identified.  Smaller  projects  are  often  grouped  together  to 
form  larger  development  programs,  but  they  may  also  stand-alone.  Some  Program  Offices  have 
a  large  number  of  project-level  development  efforts.  One  Program  Office  for  a  mature  aircraft 
system  followed  over  66  different  projects.  Those  projects  can  have  separate  schedules  and 
budgets,  and  are  often  accomplished  through  different  contractors  or  teams  within  a  primary 
contractor.  One  example  of  a  separate  development  project  within  a  larger  program  is  the  Crew 
Training  System  for  the  Joint  STARS  Aircraft  Program.  But  not  all  programs  are  run  as  separate 
efforts.  For  highly  integrated  development  efforts  such  as  the  F-22,  it  is  more  difficult  to 
separate  many  of  the  development  efforts  from  the  central  project.  In  that  case,  it  may  be  more 
meaningful  to  look  at  the  entire  program  as  a  single  development  effort. 

Another  alternative  is  to  look  at  the  development  phases-phase  I:  demonstration  and 
validation,  and  phase  II:  engineering  and  manufacturing  development-within  a  project.  But 
many  projects  have  only  a  single  development  phase,  and  the  activities  in  phases  I  and  II  are 
often  not  distinct  or  distinguishable.  A  long  first  phase  can  lead  to  a  short  second  phase,  with  no 
effect  on  actual  development  time. 

I  decided  that  the  appropriate  level  for  this  research  is  the  individual  project  level  and 
program  level,  where  the  program  could  be  considered  a  single  large  development  project.  This 
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level  provides  the  most  information  about  the  schedule  development  process  and  its  impact  on 
the  eventual  project  schedule. 

C.  Depth  versus  Breadth  of  the  Research 

The  issue  of  depth  versus  breadth  of  research  is  important  given  the  limited  resources  and 
time  for  research  projects.  An  in-depth  study  of  a  single  area  would  allow  for  a  full  description 
of  the  process  but  might  leave  out  related  areas  that  may  play  a  significant  role.  The  issue  of 
depth  versus  breadth  of  this  research  has  been  partially  addressed  by  limiting  this  research  to  the 
area  of  the  schedule  development  process  and  its  effects  on  product  development  time.  Within 
the  specific  area  of  development  of  the  project  schedule,  the  research  is  broader  and  covers  a 
range  of  potential  issues. 

The  question  of  depth  versus  breadth  also  applies  to  the  individual  projects  studied.  One 
can  study  a  few  projects  in-depth  or  a  large  number  of  projects  in  less  depth.  Research 
conducted  as  part  of  the  Lean  Aircraft  Initiative  has  relied  on  both  broad-based  surveys  and  in- 
depth  case  studies,  depending  on  the  objectives  of  the  research. 

The  overall  intent  of  the  Lean  Aircraft  Initiative  and  of  this  research  is  not  simply  to 
identify  issues  in  defense  acquisition  and  the  defense  industry,  but  to  improve  the  defense 
acquisition  system  and  the  defense  industry.  For  the  results  of  this  research  to  result  in 
widespread  change,  it  must  be  possible  to  draw  broad-based  conclusions  and  recommendations 
that  apply  to  a  large  number  of  projects,  not  just  a  few  individual  cases.  Because  of  the  unique 
nature  of  each  development  effort,  this  implies  that  in-depth  research  on  even  a  few  carefully 
selected  projects  may  not  be  representative  of  the  issues  affecting  other  projects.  Without  a  large 
number  of  case  studies,  it  may  be  difficult  to  demonstrate  convincingly  that  the  issues  in  the  case 
studies  are  applicable  to  a  broader  set  of  projects.  Others  research  on  schedules  based  on  a 
limited  number  of  case  studies  has  not  had  cut  development  times.  I  personally  observed  this 
result  early  in  my  research,  when  I  presented  data  on  one  current  project.  My  early  conclusions 
were  discounted  as  unique  to  a  special  case  and  not  applicable  to  a  broader  set  of  projects.  No 
set  of  current  project  could  be  found  that  would  be  widely  accepted  as  representative  of  the 
acquisition  process.  Each  project  encompasses  unique  circumstances  and  issues.  Thus  I  decided 
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that  to  have  the  desired  impact  on  acquisition  policy  and  practice,  I  must  evaluate  a  wide  range  of 
projects  to  determine  influences  affecting  all  of  them,  and  discover  which  influences  affect  which 
projects. 


D.  Evaluating  Available  Schedule-Based  Data 

When  trying  to  determine  what  scheduling  data  exist,  particularly  for  individual  projects, 

I  quickly  found  that  very  little  information  was  available.  I  did  identify  and  evaluate  many 
potential  sources  of  information.  These  are  described  in  Table  5-1  below. 

I  did  not  locate  any  database  or  list  of  current  project-level  development  efforts,  let  alone 
one  that  contained  the  needed  schedule-based  information.  The  data  that  could  be  located  were 
often  based  on  defense  acquisition  executive  summaries  (DAES),  acquisition  program  baselines 
(APB),  and  selected  acquisition  reports  (SAR).  Data  available  through  these  sources  are  limited 
to  major  defense  acquisition  programs.  None  of  the  databases  identified  contained  more  than  the 
resulting  project  or  program  schedules.  The  database  for  project  schedules  developed  by  The 
RAND  Corporation  from  the  selected  acquisition  reports  was  most  useful,  because  it  was  in  a 
format  that  allowed  for  analysis. 

The  available  data  did  not  provide  any  basis  for  discovering  what  factors  determine  the 
length  of  schedule  estimates.  Justification  for  schedule  estimates  is  not  required,  unlike  for  cost 
estimates.  To  conduct  meaningful  research  on  the  schedule  development  process,  I  would  clearly 
have  to  collect  original  data. 
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Acquisition  Program  Baselines:  A  database  of  program  baselines  is  maintained  by  the  SPELL  OUT 
API  of  the  Office  of  the  Secretary  of  Defense  (OSD)  for  all  ACAT I  programs.  These  baselines  consist 
of  cost,  schedule,  and  performance  parameters.  The  data  contained  in  the  files  consist  only  of  the 
resulting  schedules  and  do  not  contain  any  rationale  or  justification  for  the  duration  of  the  project.  This 
information  can  provide  information  on  project  slip  but  not  insight  into  the  schedule  development 
process. 

Defense  Acquisition  Executive  Summary:  DAES  is  a  quarterly  report  each  major  program  submits  to 
OSD  detailing  the  progress  of  a  particular  program.  The  reports  are  maintained  in  paper  form  but  are 
being  moved  to  electronic  files.  The  reports  are  required  for  all  ACAT  I  programs.  Program  reporting 
does  not  occur  until  after  projects  begin,  which  eliminates  the  possibility  of  collecting  data  on  the 
planning  phases  that  lead  to  initiation  of  a  project. 

Cost  Accounting  Reporting  System:  The  CARS  database  includes  information  pulled  from  the  DAES 
reports  that  tracks  cost  and  schedule  progress.  This  database  contains  information  on  each  major  defense 
contract,  but  does  not  include  early  information  on  schedule  development. 

Selected  Acquisition  Reports:  SAR  reports  are  the  official  reports  OSD  submits  to  Congress  detailing 
progress  on  major  programs.  These  are  similar  to  the  DAES  reports  but  occur  annually.  They  do  not 
appear  to  be  useful  in  analyzing  drivers  in  the  schedule  development  process,  but  they  can  indicate 
whether  programs  are  adhering  to  their  planned  schedules.  Several  groups  have  used  the  SAR  reports  for 
analyzing  cost  and  schedule  performance. 

RAND  Selected  Acquisition  Database:  The  RAND  Corp.  has  developed  a  database  from  the  selected 
acquisition  reports  that  allows  easier  access  to  and  analysis  of  the  data.  This  database  is  limited  to  major 
defense  acquisition  programs  and  does  not  shed  light  on  the  schedule  development  process,  only  on  the 
resulting  planned  and  achieved  schedules. 

Defense  Budget  Databases:  Defense  budget  databases  are  available  for  analysis  at  the  Pentagon.  These 
databases  could  show  the  amounts  allocated  for  particular  programs  in  a  Future  Year  Defense  Plan, 
allowing  them  to  be  tracked  over  time.  But  although  data  on  individual  projects  may  be  available  in 
some  cases,  a  majority  of  development  efforts  would  be  contained  within  major  programs  and  not  broken 
out  a  individual  budget  items.  The  data  are  thus  difficult  to  align  with  individual  development  efforts. 
Many  such  efforts  receive  money  from  different  funding  line  items,  and  tracking  a  entire  project  budget 
is  not  practical  given  the  complexity  of  using  the  database. 

Milestone  Approval  Database:  For  many  years.  Air  Force  headquarters  had  maintained  a  database  of 
major  milestone  decisions,  but  the  office  stopped  maintaining  the  database  several  years  ago  owing  to 
changes  in  the  organization  and  budget  cuts.  This  database  included  projected  milestones  but  did  not 
provide  information  on  development  or  justification  of  the  project  schedule. 

Requirements  Document  Databases:  Air  Force  headquarters  maintains  a  database  of  operational 
requirement  documents  and  mission  needs  statements.  These  documents  include  some  of  the  earliest 
actions  associated  with  projects  but  do  not  provide  a  significant  amount  of  scheduling  information.  Most 
of  the  information  is  in  paper  format,  with  only  a  limited  amount  entered  into  a  database. 

Program  Office  Records  -  Program  Office  records  contain  little  information  on  development  of  initial 
project  schedules. 


Table  5-1:  Available  Schedule-Related  Information 
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E.  Research  and  Data  Collection  Methods 

Because  a  large  number  of  projects  needed  to  be  studied  to  provide  a  clear  understanding 
of  the  schedule  development  process,  and  the  required  data  did  not  exist,  a  significant  data 
collection  effort  was  clearly  required.  Several  methods  and  considerations  were  taken  into 
account  in  determining  the  best  method  for  collecting  the  needed  data.  Traditionally,  research 
into  defense  acquisition  issues  has  primarily  focused  on  case  studies  and  applied  lessons  from 
one  project  to  future  projects.  This  process  is  complicated  by  great  differences  among  projects 
and  the  surrounding  circumstances.  Ascertaining  which  lessons  apply  to  different  projects,  and 
in  what  cases,  is  difficult  using  the  case  study  methodology. 

Several  standard  research  methods  were  evaluated  for  collecting  information  on  the  schedule 
development  process.  Those  methods  included  interview-based  research,  archival  research,  case 
study-based  research,  and  survey  research.  Each  process  has  advantages  and  disadvantages,  and 
is  appropriate  in  different  circumstances.  A  short  discussion  of  the  issues  entailed  in  selecting 
the  research  method  follows.  The  processes  and  a  short  description  are  included  in  Table  5-2  and 
Table  5-3. 


Interview-Based  Research:  Interviews  of  key  actors  are  an  effective  method  for  gaining  a  broad 
understanding  of  a  process.  But  interviews  do  not  supply  the  amount  or  type  of  data  needed  to 
convince  policy  makers  within  the  Department  of  Defense  to  change  course.  Interview  data 
strongly  depend  on  who  is  interviewed  and  are  open  to  significant  interpretation  on  the  part  of 
both  interviewee  and  interviewer.  Groups  with  senior  members  such  as  the  Defense  Science 
Board  often  rely  on  interview-related  research  to  change  policies,  but  a  less  senior  researcher 
relying  on  that  approach  was  unlikely  to  be  successful.  Thus  I  conducted  many  interviews  with 
people  working  at  the  Pentagon,  Program  Offices,  and  defense  contractors  to  establish  the  scope 
of  the  research  and  identify  the  issues  involved,  but  did  not  rely  on  them  as  the  primary  source  of 
data  for  this  research. 
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Archival-Based  Research:  Archival-based  research  relies  on  reviewing  program  documents  and 
gathering  the  required  information  from  them.  A  review  of  Program  Office  records  was 
eliminated  as  the  primary  research  method  owing  to  the  lack  of  recorded  information  on  the 
processes  used  to  develop  initial  project  schedules.  Other  researchers  reported  similar  findings.84 


Case  Study-Based  Research:  Case  studies  provide  an  opportunity  to  examine  a  few  projects  in 
depth,  allowing  for  through  analysis  of  the  processes  entailed  in  those  programs.  The  major 
issue  is  whether  the  results  from  a  few  programs  adequately  represent  a  majority  of  projects,  and 
whether  circumstances  surrounding  those  projects  are  unique.  The  applicability  of  the  results  is 
highly  dependent  on  the  projects  studied,  and  the  number  of  case  studies  that  might  have  been 
undertaken  was  limited  by  resource  and  time  constraints. 

Nevertheless,  I  did  conduct  an  in-depth  case  study  as  well  as  several  more  limited  case 
studies  to  identify  and  frame  the  issues.  This  research  was  invaluable  in  developing  the  surveys. 


Survey-Based  Research:  A  survey  method  allows  for  study  of  a  larger  number  of  projects  but 
not  at  the  depth  of  a  case  study.  The  types  of  questions  that  can  be  asked  and  the  total  length  of 
the  survey  limit  survey-based  research.  But  proper  sampling  methods  and  a  large  number  of 
respondents  can  ensure  that  the  results  accurately  reflect  the  population  of  projects.  Such  surveys 
can  reveal  the  general  nature  of  the  process  used  to  develop  project  schedules.  Relying  on  data 
collected  from  an  array  of  defense  development  projects  provided  a  more  robust  database  from 
which  to  draw  conclusions  and  base  recommendations. 


For  this  research,  a  broad-based  survey  of  current  development  projects  offered  a  better 
opportunity  to  specify  the  general  nature  of  the  process.  The  broad-based  survey  ensured  that  the 
results  would  be  representative  of  the  schedule  process  as  a  whole  and  not  any  circumstances 
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surrounding  a  particular  project.  The  survey  method  was  augmented  primarily  by  interviews, 
case  studies,  and  literature  reviews  to  understand  the  issues  involved  both  before  developing  the 
surveys  and  during  analysis  of  the  results.  Discussion  groups  and  interviews  were  conducted  at 
product  centers  and  within  the  Lean  Aircraft  Initiative  to  review  and  help  interpret  the  survey 
results,  to  gain  a  full  understanding  of  their  meaning. 


Method 

Advantages 

Disadvantages 

Interview 

Research/Discussion 

Groups 

Easy  to  conduct 

Does  not  rely  on  extensive 
data  collection 

Captures  experience  of  others 

Results  dependent  on  the  experiences 
of  those  interviewed 

Difficult  to  rectify  difference  of 
opinion 

Unlikely  to  change  policies 

Archival  Research 

Data  recorded  at  time  of 
events 

Limited  schedule-related  data 
available.  Difficult  to  understand 
issues  involved.  Poor  documentation 
of  development  of  initial  project 
schedules. 

Case  Study  Research 

Allows  for  in-depth 
examination  of  schedule 
development  issues  on 
several  projects 

Limited  number  of  potential  projects 

Each  project  has  special  circumstances 
that  may  make  drawing  broader 
lessons  difficult 

Survey-Based 

Research 

Allows  for  a  large  number  of 
projects  to  be  studied 

Allows  statistical  analysis  of 
results 

Limited  insight  into  each  program 
Limited  to  size  and  scope  of  survey 
Limited  to  depth  of  questions 

Literature  Research 

Many  sources  on  commercial 
product  development  efforts 

Few  authoritative  sources  on 
developing  defense  products 

No  literature  on  developing  project 
schedules  for  defense  systems 

Lessons  from  commercial  projects 
may  not  apply  to  military  projects 

Table  5-2:  Summary  of  Advantages  and  Disadvantages  of  Various  Research  Methods 
Considered  or  Used  in  this  Research 


84  Drezner  and  Smith  Rand  Dec  1990;  Hazelton  and  Tofler  AFIT  1993;  Boyd  and  Mundt  AFIT  1993;  Harmon  and 
Ward  1990;  Tyson,  Nelson,  Om,  and  Palmer  1989. 


5.  Research  Methodology 
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Method 

Initial  Research  Scoping 

Schedule  Development 
Research 

Follow-up  Analysis 
and  Interpretation 

Interviews  and 

Discussion 

Groups 

Many  interviews  were 
conducted  with  a  wide 
range  of  people  from 
government,  industry,  and 
academia  to  understand 
issues  related  to 
development  time 

Used  to  develop  and  test 
the  survey.  Ensured 
data  availability 

Interviews  and  group 
discussions  were  used 
to  understand  survey 
results  and  screen 
recommendations 

Archival 

Research 

Used  to  locate  available 
data.  Few  useful  data 
found  at  Program  Offices 

Case  Study 

Research 

A  major  case  study  and 
several  smaller  case  studies 
were  conducted  to 
understand  the  issues 
involved  in  developing 
project  schedules 

Results  of  case  studies 
used  to  develop  surveys 

Survey-Based 

Research 

Three  surveys  used  to 
collect  data  on  projects 

Analysis  based  on 
database  from  surveys 

Literature 

Research 

An  extensive  number  of 
books,  papers,  articles,  and 
course  materiel  were 
reviewed  to  set  scope  and 
frame  for  the  research 

Used  to  identify 
potential  survey 
questions  and  wording 

Table  5-3:  Research  Methods  Used  during  Various  Stages  of  Research 
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Chapter  Summary 

This  research  selected  an  exploratory  vice  a  hypothesis  driven  research  approach.  Little 
previous  conclusive  research  and  widely  varying  opinions  among  defense  leaders  indicated  that  a 
broader  study  of  the  schedule  related  issues  was  warranted  than  hypothesis  driven  research  would 
allow.  The  appropriate  level  of  analysis  for  this  research  was  selected  as  the  individual  project  as 
it  was  felt  that  only  at  the  individual  project  level  could  the  factors  influencing  a  projects 
schedule  and  the  impacts  of  that  schedule  be  determined.  An  analysis  of  the  available  schedule 
related  data  indicated  that  significant  data  collection  would  be  required.  To  collect  the  data 
survey-based  method  was  selected  as  the  appropriate  method  as  it  allowed  for  a  wide  array  of 
projects  to  be  studied. 


Chapter  6 


Project  Surveys:  An  Overview  of  Program  Office, 
Contractor,  and  Pentagon  Surveys _ 


A.  Survey  Objectives 

The  objectives  of  the  surveys  for  this  research  were  to  document  and  describe  the 
schedule  development  process  and  its  impact  on  eventual  product  development  times.  The 
surveys  did  this  by  gathering  information  about  projects  representing  a  cross  section  of  defense 
aerospace  products.  Two  of  the  surveys  for  this  research  were  conducted  jointly  with  Dr.  Eric 
Rebentisch  of  the  Lean  Aircraft  Initiative,  who  was  studying  issues  associated  with  managing 
programs  under  funding  instability. 

B.  Overview  of  the  Three  Surveys 

Prior  to  developing  the  surveys,  I  identified  the  key  players  that  influence  schedules,  and 
the  types  of  information  needed  to  characterize  the  schedule  development  process.  I  recognized 
that  respondents’  perspectives  might  depend  on  the  organization  to  which  they  belong  and  the 
level  from  which  they  view  the  development  effort.  To  obtain  a  comprehensive  look  at  the 
schedule  development  process  across  various  organizations,  a  multi-level  set  of  surveys  was 
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chosen.  These  surveys  were  conducted  at  the  Pentagon,  the  Program  Offices,  and  the  defense 
contractors. 

The  three  surveys  were  designed  to  collect  information  on  the  development  of  project 
schedules  and  the  project’s  adherence  to  that  schedule.  The  surveys  were  structured  to  gather  the 
information  readily  available  to  people  working  in  the  different  organizations.  The  three 
overlapping  surveys  were  also  designed  to  provide  either  corroborating  or  supporting  responses 
among  organizations,  and  to  identify  and  highlight  the  differences. 

To  obtain  information  on  individual  projects,  the  surveys  targeted  project  managers  in  the 
Program  Offices,  project  managers  in  prime  contractor  companies,  and  program  element 
monitors  or  action  officers  in  the  Pentagon.  These  people  were  in  a  position  to  possess  the  most 
knowledge  and  understanding  of  the  schedule  development  process,  as  they  make  decisions  or  at 
least  recommendations  for  their  projects  within  their  organizations.  The  surveys  were  time 
phased  to  allow  me  to  identify  additional  areas  of  interest,  add  questions,  and  change  wording  on 
subsequent  surveys.  The  surveys  were  completed  over  a  one-and-a-half-year  period.  The 
Program  Office  survey  was  mailed  in  January  1996  and  returned  through  June  1996.  The 
Contractor  survey  was  mailed  in  June  1996  and  returned  through  October  1996.  The  Pentagon 
survey  was  mailed  in  January  1997  and  returned  through  May  1997.  The  difference  in  timing  the 
surveys  did  not  affect  the  results,  as  many  of  the  projects  have  been  in  development  for 
significantly  longer  periods,  and  the  time  required  to  change  the  system  is  significantly  longer 
than  one  year. 


C.  Survey  Development 

A  literature  search  was  conducted  to  uncover  factors  affecting  both  defense  and 
commercial  product  development.  A  list  of  potential  questions  was  developed  to  cover  the  range 
of  issues  associated  with  developing  project  schedules. 


6.  Project  Surveys:  An  Overview  of  Program  Office,  Contractor,  and  Pentagon  Surveys 
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The  first  survey  developed  was  the  Program  Office  survey,  designed  to  explore  issues 
associated  with  both  schedules  and  program  instability.85  Several  different  types  of  questions 
were  created  to  gather  factual  and  perceptual  data  from  respondents.  Questions  to  gather  factual 
data  requested  specific  information  believed  to  be  readily  available  to  respondents  based  on 
pretest  results. 

Some  of  the  needed  information  concerned  people’s  perceptions  and  interpretations  of 
events  and  their  influence.  Thus  while  most  questions  were  specific  to  make  data  analysis  easier 
and  force  concise  responses,  some  open-ended  questions  were  included  to  give  respondents  an 
opportunity  to  explain  specific  circumstances.  Different  scales  were  used  depending  on  the 
nature  of  the  questions  and  the  information  desired.  In  all  cases,  the  questions  were  structured  to 
collect  the  required  information  and  to  ease  the  burden  on  respondents.  The  text  of  the  surveys 
can  be  found  in  the  appendix.  Examples  of  questions  and  the  method  used  to  analyze  the  results 
can  be  found  in  Chapter  7. 

The  surveys  were  organized  into  major  sections  that  focused  on  the  characteristics  of 
individual  projects,  their  costs  and  cost  growth  of  projects,  scheduled-related  information,  and 
personal  information  on  respondents.  This  separation  is  more  evident  in  the  later  surveys,  as 
significant  learning  occurred  during  collection  of  the  data  from  the  first  survey. 

Each  survey  was  reviewed  and  pre-tested  in  several  settings.  Throughout  their 
development,  draft  copies  were  assembled  and  circulated  among  fellow  researchers  for  comment. 
Early  drafts  were  then  reviewed  by  a  sample  of  program  managers  at  Hanscom  Air  Force  Base 
(AFB)  in  an  interview  format.  After  their  comments  were  incorporated,  the  survey  was  given  in 
final  draft  form  to  15  project  managers  at  Hanscom  AFB.  These  program  managers  provided 
both  data  and  comments  on  the  survey.  They  also  estimated  the  time  respondents  would  need  to 
complete  the  survey  (between  1  and  3  hours,  depending  on  their  direct  knowledge  of  a  project), 
and  a  breakout  of  the  time  required  for  each  section.  These  people’s  comments  and  suggestions 
on  both  content  and  wording  were  very  helpful  and  incorporated  into  the  final  survey. 


85  The  Program  Office  Survey  and  the  Contractor  Survey  were  conducted  jointly  with  Dr  Eric  Rebintisch  with  the 
Lean  Aircraft  Initiative.  Dr  Rebintisch  was  studying  Program  Instability  in  Defense  Projects.  His  research  is 
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Subsequent  surveys  used  questions  similar  to  the  Program  Office  survey  and  were  reviewed  by 
the  respective  organizations  and  a  representative  group  of  potential  respondents  prior  to  their 
distribution. 


D.  Survey  Limitations 

The  survey  method  has  a  number  of  limitations:  The  number  and  content  of  questions 
that  can  be  asked  are  limited,  and  the  length  of  the  survey  must  be  limited  to  avoid  affecting  the 
response  rate  significantly.  The  information  collected  is  also  dependent  on  the  respondents’ 
knowledge  of  events  in  question,  their  ability  to  find  the  information,  and  their  willingness  to 
provide  that  information.  The  accuracy  and  completeness  of  the  responses  cannot  be  controlled. 

A  target  limit  was  placed  on  the  number  of  pages  of  the  survey  based  on  survey  literature. 
Prior  experience  within  LAI  also  indicated  that  the  response  rate  would  decline  proportionally 
with  the  length  of  the  survey.  Every  effort  was  made  to  limit  the  questions  to  areas  deemed 
necessary  to  describing  the  scheduling  process  and  instability  issues.  The  length  of  the 
contractor  survey  was  limited  even  further  at  the  direction  of  the  sponsoring  organizations. 

Surveys  are  also  limited  in  their  ability  to  measure  prior  events,  as  people’s  recollections 
may  fade  and  later  events  may  affect  interpretations.  This  is  a  problem  with  all  research.  A 
decision  was  made  to  use  the  retrospective  approach  because  of  excessive  the  time  entailed  in  in- 
si  tu  research.  There  was  no  option  other  than  to  rely  on  respondents’  recollection  and 
understanding  of  events. 

Surveys  are  also  limited  by  the  willingness  of  respondents  to  provide  information.  To 
surmount  concerns  about  the  use  of  the  information,  strict  assurances  of  confidentiality  were 
provided.  I  promised  that  the  information  would  be  masked  if  it  became  apparent  that  an  outside 
person  could  connect  the  responses  with  a  specific  project. 


available  through  the  Lean  Aircraft  Initiative.  “Managing  Under  Program  Instabilities”  LAI  Draft  report. 


6.  Project  Surveys:  An  Overview  of  Program  Office,  Contractor,  and  Pentagon  Surveys 
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E.  Organizations  Surveyed 

When  determining  which  organizations  to  survey  for  the  Program  Office  survey,  Dr. 
Rebentisch  and  I  decided  to  include  those  focused  on  the  entire  selection  of  aerospace  products 
across  all  three  services.  Previous  studies  had  been  severely  limited  in  the  number  of  projects, 
which  in  turn  limited  the  use  of  statistical  analysis.  Surveying  all  aerospace  development 
projects,  and  not  relying  on  any  predetermined  selection  criteria,  avoided  any  selection  bias.  The 
projects  included  would  depend  only  on  the  cooperation  of  the  organizations  and  individual 
respondents. 

Air  forces  organizations  identified  included  groups  from  all  three  services.  The  Air  Force 
organizations  were  the  Aeronautical  Systems  Center  at  both  Wright  Patterson  Air  Force  Base  and 
Eglin  Air  Force  Base,  the  Electronic  Systems  Center  at  Hanscom  Air  Force  Base,  and  the  Space 
and  Missile  Center  at  Los  Angeles  Air  Force  Base.  The  Navy  organization  primarily  associated 
with  developing  aerospace  products  is  the  Naval  Air  Systems  Command  (NAVAIR).  Army 
organizations  associated  included  the  Army  Aviation  and  Troop  Command  (ATCOM)  and  the 
Army  Tactical  Missile  and  Missile  Defense  programs  at  Redstone  Arsenal.  Other  Air  Force 
projects  run  from  the  Air  Force  Air  Logistic  Centers  were  not  primarily  development  efforts  and 
were  not  included  in  the  surveys.  Often  an  associated  Program  Office  within  the  various 
development  centers  runs  many  of  the  development  projects  for  mature  systems,  primary 
responsibility  for  which  is  formally  transferred  to  logistics  centers.  We  decided  that  excluding 
the  logistics  support  centers  would  not  compromise  the  representative  nature  of  the  project 
samples,  and  would  not  impact  the  integrity  of  the  survey. 

The  population  of  development  efforts  included  a  wide  range  of  project  sizes  and  a  wide 
variety  of  systems.  Since  the  goal  was  to  describe  the  schedule  development  process  across  the 
entire  range  of  projects,  it  was  important  to  include  the  full  range  within  the  target  survey 
sample. 

To  develop  a  list  of  ongoing  projects  and  their  project  managers,  we  contacted  all  the 
development  centers.  Many  of  the  product  centers  did  not  maintain  a  list  of  ongoing 
development  projects  but  instead  provided  lists  of  the  organizations  involved.  No  centralized  list 
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could  be  found,  for  example,  of  ongoing  development  projects  at  the  Air  Force  level,  the  center 
level,  or  even  the  Program  Office  level.  In  many  cases,  it  was  necessary  to  call  each  Program 
Office  directly  and  speak  with  various  team  leaders  to  identify  the  set  of  ongoing  projects. 
Despite  significant  efforts,  a  suitable  list  of  projects  was  not  developed  from  the  Air  Force  Space 
and  Missile  Center,  and  its  participation  in  the  study  was  dropped.  In  several  cases  the 
organizations  requested  that  the  smallest  projects  be  excluded  (Aeronautical  Systems  Center  and 
ATCOM),  or  provided  only  the  major  programs  (NAVAIR).  This  resulted  in  a  higher  percentage 
of  larger  programs  in  the  sample  than  may  occur  in  the  population  of  projects  as  a  whole.  Both 
the  Aeronautical  Systems  Center  and  the  Electronic  Systems  Center  provided  information  on  a 
large  number  of  projects  and  significant  support  in  ensuring  that  the  surveys  were  completed.  As 
a  result,  a  significant  percentage  of  the  returned  surveys  represent  ASC  and  ESC  programs.  As 
will  be  shown  later,  there  was  no  significant  difference  in  survey  results  between  larger  and 
smaller  projects  or  among  different  product  centers.  Thus  there  did  not  appear  to  be  any  bias 
effect  stemming  from  the  surveyed  population.  In  total  430  ongoing  projects  were  identified  at 
the  Program  Office  level,  and  surveys  were  mailed  to  all  430  project  teams. 

The  contractor  survey  relied  on  the  willingness  of  the  companies  to  participate  and  our 
ability  to  identify  suitable  projects  and  appropriate  respondents.  Two  methods  were  used  to 
identify  projects  for  this  survey.  The  Program  Office  survey  asked  the  program  managers  to 
identify  the  contractor  program  manager  and  provide  an  address.  And  companies  participating  in 
the  Lean  Aerospace  Initiative  helped  identify  projects.  The  sampled  population  included  all 
projects  identified  in  the  Program  Office  survey  and  all  projects  identified  by  LAI  associated- 
contractors  that  elected  to  participate. 

An  effort  was  again  made  to  identify  every  significant  development  project  and  send  a 
survey  to  each.  No  effort  was  made  to  select  or  exclude  specific  projects  from  the  survey  based 
on  performance.  In  addition,  we  agreed  to  shield  from  people  other  than  the  researchers  any 
identifying  information  that  would  allow  responses  to  be  associated  with  a  specific  project.  In 
total,  250  individual  projects  and  their  program  managers  were  identified  for  the  contractor 
survey. 


6.  Project  Surveys:  An  Overview  of  Program  Office,  Contractor,  and  Pentagon  Surveys 
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For  the  Pentagon  survey  we  adhered  to  the  same  philosophy  of  identifying  the  entire 
population  of  interest  and  sending  surveys  to  all.  The  Office  of  the  Assistant  Secretary  of  the  Air 
Force  for  Acquisition  provided  a  list  of  all  Air  Force  program  element  monitors  and  associated 
action  officers.  As  we  attempted  to  identify  Army  and  Navy  program  element  monitors,  we 
discovered  that  those  services  consolidated  their  program  elements,  providing  less  insight  into 
individual  development  projects.  We  therefore  decided  not  to  include  Navy  and  Army  program 
element  monitors  in  the  Pentagon  survey. 

The  list  of  program  elements  and  program  element  monitors  was  screened  to  eliminate 
the  elements  that  clearly  did  not  oversee  any  development  efforts.  Surveys  were  mailed  to  all 
225  program  element  monitors  and  action  officers  who  might  oversee  development  efforts. 
Approximately  100  of  the  225  program  element  monitors  responded  that  they  did  not  oversee 
any  appropriate  development-related  activities. 

Since  many  of  the  program  element  monitors  oversaw  several  program  elements 
containing  many  development  efforts,  each  monitor  and  action  officer  was  asked  to  answer  only 
one  survey  for  a  single  project  with  which  they  were  most  familiar.  Having  the  program  element 
monitors  complete  more  than  one  survey  was  not  seen  as  practical  nor  likely  to  be  successful. 
While  this  allowed  less  control  over  the  population  of  projects  and  may  have  allowed  for  some 
selection  bias  on  the  part  of  the  monitors,  the  returned  surveys  matched  the  general 
demographics  of  the  other  two  surveys.  We  judged  that  data  obtained  on  projects  with  which  the 
monitors  were  most  familiar  would  be  of  higher  quality  and  more  accurate  than  responses  on  a 
project  they  were  only  peripherally  aware  of.  Allowing  the  monitors  to  select  the  project  was 
assumed  to  boost  the  response  rate. 

We  did  make  a  largely  unsuccessful  attempt  to  direct  program  element  monitors  to 
projects  already  included  in  the  Program  Office  and  contractor  surveys,  to  provide  direct  overlap. 
But  the  limited  number  of  respondents  and  less-than-optimal  response  rates  resulted  in  a  low 
number  of  overlapping  surveys.  Only  8  surveys  pertained  to  projects  in  both  the  Pentagon  and 
Program  Office  surveys — not  enough  for  meaningful  statistical  analysis.  A  visual  inspection  of 
the  responses  did  not  reveal  any  large  differences  between  Pentagon  and  Program  Office 
responses  on  the  same  projects. 


144  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


In  total,  905  potential  respondents  were  identified  at  the  Program  Offices,  the  defense 
contractors,  and  the  Pentagon.  Surveys  were  identified  by  project  and  addressed  to  the  project 
managers  or  program  element  monitor  of  the  specific  project  identified.  Program  Element 
Monitors  with  several  projects  were  mailed  a  single  survey  and  asked  to  select  a  single  project  as 
described  earlier.  A  survey  number  tracked  each  survey. 


F.  Response  Rate  and  Efforts  to  Improve  It 

Prior  to  and  following  the  distribution  of  each  survey,  significant  efforts  were  made  to 
improve  the  response  rate.  Efforts  taken  prior  to  the  Program  Office  survey  included  obtaining 
senior  management  support  at  the  product  centers  and  among  the  Air  Force  acquisition  staff.  Mr. 
Blaise  Durante,  deputy  assistant  secretary  of  the  Air  Force  for  management  policy  and  program 
integration,  signed  a  cover  letter  directing  the  Program  Offices  to  complete  the  survey. 

A  multi-step  process  was  used  to  boost  the  response  rate  following  distribution  of  the 
Program  Office  surveys.  A  follow-up  letter  to  each  recipient  was  sent,  succeeded  by  phone  calls. 
A  few  weeks  later,  a  message  was  sent  from  the  commander’s  office  at  each  product  center 
directing  recipients  to  return  the  surveys.  A  few  weeks  later  a  list  of  people  that  had  returned  the 
surveys  and  those  that  had  not  was  provided  to  the  command  sections  of  each  center,  and  a 
second  message  from  the  commander’s  executive  officer  was  sent  to  each  recipient.  Later 
additional  individual  calls  were  placed  to  individuals  to  encourage  them  to  return  the  completed 
surveys.  Replacement  surveys  were  mailed  to  those  who  had  lost  them,  and  additional  surveys 
were  provided  to  the  individuals  requesting  them  when  contacted. 

The  efforts  to  boost  the  response  rate  did  not  have  a  dramatic  effect  on  the  number  of 
surveys  returned.  Similar  efforts  were  made  for  the  contractor  and  Pentagon  surveys. 

The  final  response  rates  for  the  three  surveys  were  35  percent  for  the  Program  Office 
survey,  42  percent  foi;  the  contractor  survey,  and  49  percent  for  the  Pentagon  survey.  This 
excludes  responses  from  the  Program  Element  Monitors  that  indicated  that  they  did  not  have  any 
development  projects  within  their  program  elements. 
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G.  Respondent  Profiles 

Between  the  three  surveys,  317  completed  usable  surveys  were  received:  151  Program 
Office  surveys,  104  contractor  surveys,  and  62  Pentagon  surveys.  Other  surveys  and  responses 
received  but  incomplete  or  lacking  information  were  not  included  in  the  analysis.  The  complete 
list  of  projects  whose  managers  responded  is  provided  at  the  end  of  the  chapter  in  Tables  6-1,6- 
2,  and  6-3. 

The  projects  represented  in  the  responses  can  be  characterized  in  many  different  ways, 
including  by  cost,  acquisition  phase,  product  type,  production  cost,  and  year  the  project  was 
initiated.  That  information  is  provided  to  show  the  types  of  projects  from  which  the  data  are 
culled.  Because  no  description  of  the  total  population  of  development  projects  is  available,  no 
comparison  to  the  total  population  can  be  made.  The  consistency  of  the  distribution  and  project 
characteristics  across  the  three  separate  surveys  provides  some  evidence,  although  not  proof,  that 
the  surveys  represent  a  reasonable  cross  section  of  ongoing  Air  Force  development  efforts. 

The  respondents  oversee  different  types  of  projects,  from  aircraft-related  systems  through 
software-dominated  systems.  Figure  6-2  below  shows  the  distribution  across  different  types  of 
systems.  The  responses  cover  a  significant  number  of  projects  from  each  category,  with  a  large 
number  from  aircraft  avionics  programs.  The  absence  of  spacecraft-related  systems  in  the 
Program  Office  survey  and  the  contractor  survey  is  due  primarily  to  the  exclusion  of  the  Air 
Force  Space  and  Missile  Center  from  that  survey,  and  to  the  focus  of  the  companies  involved  in 
the  Lean  Aerospace  Initiative  at  that  time. 

The  projects  can  also  be  grouped  into  acquisition  categories  (AC AT) .defined  by  the  size 
of  the  project.  ACAT I  projects  are  the  largest,  with  over  $355  million  in  development  costs  and 
$2.1  billion  in  production  costs.  These  are  considered  the  major  defense  acquisition  programs. 
ACAT  II  projects  have  between  $355  million  and  $140  million  in  development  costs.  ACAT  HI 
and  IV  projects  have  less  than  $140  million  in  development  costs.  There  was  no  significant 
difference  between  the  distribution  of  ACAT  levels  between  the  Program  Office  respondents  and 
Pentagon  respondents,  indicating  that  they  represented  a  similar  distribution  of  projects. 
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Comparison  of  the  distribution  of  ACAT  projects  with  the  contractor  survey  was  complicated  by 
the  large  number  of  respondents  who  did  not  indicate  the  ACAT  level  for  their  development 
effort. 


Survey 


Contractoi 


□ 


PEM 


System  Type 


Figure  6-1:  Percentage  of  Respondents  by  Survey  and  System  Type 
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ACAT  I  ACAT  II  ACAT  III 


Figure  6-2:  Distribution  of  the  Acquisition  Category  (ACAT)  of  Responding  Projects  by 
Program  Office,  Contractor,  and  Pentagon  Surveys 


Identifying  the  acquisition  phase  of  a  project  provided  another  way  to  characterize  the 
respondents.  The  largest  number  of  projects  were  in  the  engineering,  manufacturing,  and 
development  phase  or  earlier  stages.  A  significant  number  were  in  the  production  phase,  and  a 
few  were  in  the  operations/sustainment  and  closeout  phases.  Projects  covered  in  the  Program 
Office  survey  had  a  higher  probability  of  being  in  the  production  phases  than  the  projects 
covered  in  the  other  surveys. 
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Figure  6-3:  Total  Respondents  by  Acquisition  Phase 


Con  Exp  ft  Def  Eng  ft  Manf  Dev  Sustainment  Other 


Figure  6-4:  Respondents  by  Acquisition  Phase  for  Program  Office,  Contractor,  and 
Pentagon  Surveys  (Number  of  Projects  =  317) 

The  projects  surveyed  can  also  be  characterized  by  the  amount  of  technological  advance 
required  in  their  systems.  As  would  be  expected,  a  significant  number  are  based  on  either  a  new 
technological  generation  or  incremental  improvements,  and  a  smaller  number  are  based  on 
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revolutionary  development  or  little  or  no  improvement.  When  broken  out  by  survey,  the  responses 
indicate  no  significant  difference  between  the  surveys  and  expected  results  if  they  sampled  the  same 
general  population. 


Revolutionary  New  Generation  Incremental  Improvement  Little  or  None 

Product  Technology  Advance 


Figure  6-5:  Respondents  by  Amount  of  Technological  Advance  in  the  Pentagon,  Program 
Office,  and  Contractor  Surveys  (Number  of  Projects  =  317) 
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Figure  6-6:  Distribution  of  Respondents  by  Level  of  Technological  Advance  for  Program 
Office,  Contractor,  and  Pentagon  Surveys  (Number  of  Projects  =  317) 
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Another  way  to  characterize  respondents  is  by  the  cost  of  the  development  effort  they 
oversee  and  the  cost  of  the  expected  production  effort.  In  the  following  charts,  the  cost  of  the 
systems  is  grouped  by  the  order  of  magnitude  of  the  cost  of  the  development  and  production 
phases. 
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Planned  Engineering  &  Manufacturing  Development  Costs 


Figure  6-7:  Planned  Pre-EMD  and  EMD  Cost  of  the  Projects  Surveyed 


Planned  Production  Costs 


Figure  6-8:  Planned  Production  Costs  of  the  Projects  Surveyed 
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Another  way  the  projects  can  be  characterized  is  by  the  year  of  their  initiation.  Most  of 
the  projects  respondents  described  are  less  than  five  years  into  their  development  phase.  This 
indicates  that  they  represent  current  development  practices.  This  also  indicates  that  many 
respondents  had  firsthand  knowledge  of  the  information  requested,  or  enough  of  corporate 
memory  of  the  schedule  development  process  to  answer  the  questions  accurately.  Many  older 
projects  were  also  included  by  respondents,  some  dating  back  a  considerable  length  of  time.  The 
drop  off  in  the  number  of  projects  started  in  1996  may  reflect  the  timing  of  the  surveys.  The 
Program  Office  survey  was  conducted  at  the  beginning  of  1996,  so  projects  begun  after  that  date 
would  not  be  included. 


Year  of  Project  Initiation 


Figure  6-9:  Year  of  Program  Initiation  by  Survey  Respondents 
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The  apparent  bias  of  the  respondents  toward  more  recent  programs  is  preferable,  in  that 
they  represent  current  development  practices.  That  means  that  the  survey  results  indicate  how 
acquisition  processes  are  presently  working,  and  not  necessarily,  how  they  have  worked  in  the 
past. 


There  is  a  wide  spread  of  development  times.  The  average  time  from  program  initiation 
to  first  production  item  was  76  months.  A  significant  number  of  projects  reported  development 
times  in  excess  of  10  years. 


Project  Start  to  First  Production  Item  (Months) 


Figure  6-10:  Development  Time  from  Program  Initiation  to  Delivery  of  the  First 
Production  Item  for  Surveyed  Projects  (Number  of  Projects  =  225;  Average  Development 
Time=76  Months) 
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Since  there  is  no  available  list  or  characterization  of  all  ongoing  projects  that  could  be 
used  as  a  comparison,  it  is  not  possible  to  determine  if  survey  respondents  represented  a  cross 
section  of  ongoing  projects.  The  similarity  of  distribution  of  project  characteristics  among  the 
three  surveys  has  already  been  noted.  These  similar  distributions  include  the  type  of  project,  the 
size  of  projects  (ACAT),  the  acquisition  phase,  and  the  scope  of  the  technological  advance. 
These  results  provides  additional  confidence  that  the  surveys  fairly  represent  the  set  of  ongoing 
projects. 


Chapter  Summary 

The  three  surveys  were  designed  to  gather  the  necessary  data  on  the  schedule 
development  process  from  Program  Offices,  the  Pentagon,  and  defense  contractors.  The  survey 
attempted  to  survey  all  military  aerospace  defense  projects.  A  total  of  317  completed  responses 
were  received.  Characterization  of  the  responses  indicated  that  they  adequately  represent  current 
military  aerospace  development  efforts. 
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Table  6-1 

Projects  Overseen  by  Pentagon  Survey  Respondents 
(Program  Element  Monitors  and  Action  Officers) 


60K  A/C  loader 
Aces  II  Ejection  Seat 
Advanced  EHF 

Advanced  Airborne  Interceptor 
AIM-9X 

Airborne  Laser  (ABL) 

AMRAAM  P3I P3 

AW  ACS  Multi-Source  Tactical  System  B- 
1B  Defensive  Avionics  Upgrade 
B-52  Advanced  Weapons  Integration 
C-130  AW/APN-59  Radar  Replacement 
C-130J 

C-130  J  Aircraft  Annex 
C-141  All  Weather  Flight  Control 
CTAPS  5.2  Software 
CV-22 

Combat  Survivor  Evader  Locator 
Common  Missile  Warning 
Defense  Satellite  Communications 
Deliberate  Crises  Action  Planning 
Deployable  C3  Communications 
DIRECT 

EARC  Night  Vision  Imaging  System 
Environmental  Satellite  System  Evolved 
Expendable  Launch  Vehicle 
F-16NVIS  Pre-Block  40 
F-16  Onboard  Oxygen  Generator 
F-22 


GCSS-AF 

Global  Weather  Analysis  and  Prediction 
Guidance  Replacement  Program 
Have  Stare 
IMDS 

IW  Weapon  System  Vulnerability 

KC-135  Multipoint  Refueling  System 

KC-135  Simulator  Upgrade  Visual 

Joint  Air  to  Surface  Stand  Off  Missile  Joint 

Strike  Fighter 

Joint  SIGINT  Family 

JSESST 

Life  Cycle  Cost  Reduction  Initiative 
Milstar  Terminals 
NCMC-  TW/AA  System 
NPOESS  -  National  Polar  Orbiting 
Predator  UAV 

Small  Tactical  Terminal  (STT) 

SOF/Rescue 

Space  Based  Infrared  System  (SBIRS)Space 
Warfare  Center 

Strategic  War  Planning  System 
SSMIS  Tactical  Weather  Radar 
Titan  IV  Space  Boosters 
Theater  Airborne  Reconnaissance  System 
*  Tri-Service  Embedded  GPS/INS 
Wind  Corrected  Munition  Dispenser 
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Table  6-2 


Projects  Overseen  by  Contractor  Survey  Respondents  (Program  Managers) 

A-10Laste  F-22RAIU 

A2100  LDAs  F-22CNI 

ABL  -  Airborne  Laser  Program  F-22  Main  Electric  Power  Generator 

AC-130  NAV/FCO  TEST  BED  F-22  Radar 


Advanced  IRST 

Advanced  Propulsion  Materials  MA 

Affordable  Multimissile  Manufacturing 

Air  Route  Surveillance  Radar  Mode 

ALQ  131  FMS 

AN/ALQ-126B 

AN/ALQ- 144/144 A 

AFMSS 


F-l  17  Propulsion  for  C-17  Aircraft 

FI 00  Component  Improvement  Program  F414 

FADEC 

Gunship  Spares 

Horizontal  Technology  Integration  Integrated  ESM 
JDAM  -  Joint  Direct  Attack  Munition  Javelin 
Joint  Air  to  Surface  Stand  Off  Missile 
Joint  STARS 


APG-73  Phase  2  (Recce) 

Advanced  Unitary  Penatrator 
AF  Joint  Technology  Demonstrator 
Affordable  Composites  for  Propulsion 
Anti  Helicopter  Munition 
ANAPQ-174D  MMR  EMD 
ARPA  Advanced  Ceramic  Technology 
Autonomous  Intellegent  Submunition 
AV-8B  Remanufacture 
B-1B  Avionics 

B-1B  Bomber  Integrated  Logistics 

B-1B  Conventional  Mission  Upgrade 

B-1B  Production/AN/ALQ-161A 

B-l  Training  System 

B-2  Parts  EMD 

B-2  Parts  Production 

Boeing  111  Floor  Beams 

C-12  Avionics  Prototype  kits 

C-17 

C-17  Aircrew  Training  System 
C-130H2 

C-130J  Displays  and  Computers 
Combat  DF 

Commercial  Turbo  Prop 

CV-22 

DELTA  m 

E-6  Orbit  Improvement  Systems 
EF-1 1 1  System  Improvement  Program 
Exoatmospheric  Kill  Vehicle  Sensor 
F-14  Lantim  Integration 
F-15 

F-15AVTR 

F-15  APG  63(V)1  (APG-63  Radar  RM 

F-15S  to  Saudi  Arabia  FMS 

F-16 

F/A-18  E/F 


Joint  Strike  Fighter 

JPATS  -  Joint  Primary  Aircraft  Training  LANTIRN 
F-14 

Launcher  Avionics  Package 

LCA IFCS 

LMD/KP 

Longbow 

Long  Bow  Apache 

Long  Bow  FCR 

Long  Bow  Missile 

M-l  Tank  (General  Dynamics) 

MAFET  Thrust  2 
Mast  Mounted  Sight 
Milstar  Terminal  (AF) 

Missile  Tracking  System 

MQM107D 

Navy  Landing  System 

Navy  Joint  Technology  Demonstrator 

Peace  Shield 

RAH-66  Comanche 

RAH-66  Comanche  CNI 

SEA  Transmission 

Sensor  Fused  Weapon 

Sensor  Fuzed  Weapon  -  P3I  Improvement 

SLAM  (ER) 

Space  Station  Communication  System 

Space  Station  -  Power  Module 

SR-71  Elint  Production 

Tactical  Endurance  SAR 

T-1A  Trainer 

T-56  Turbo  Prop  Engine 

TAD/PNVS 

THAADS 

Tier  11+  Wing 

TLOS 

Trident  D5  Missile  TMK  Translator 


F/A-18  E/F  AM  AD  Aircraft  Mounted 

F-22 

F-22  EW 


V-22 

Wavelet  Insertion  Demonstration 
Wide  Area  Munition 
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Table  6-3 

Projects  Overseen  by  Program  Office  Survey  Respondents 
(Program/Project  Managers) 


AC-130U  Gunship 
Advanced  Interface  Control  Unit 
Advanced  Cruise  Missile 
Advanced  Strategic  and  Tactical 
Aerial  Targets 

Aircrew  Integrated  Helmet  System 
AIM-9X 

Air  Combat  Command  Training  Prog 
Air-To-Ground  Missile  Systems 
AGM-129 

Alternate  Processing  Correlation 
AMC  C2  IPS  Info  Processing 
AMRAAM  AIM-  120A/B/C/  Missile 
AN/ALE-47  Countermeasure  Dispen 
Apache 

Apache  Attack  Helicopter 
AV-8B  Remanufacture 
AWACS 
AWACS  -  Saudi 
AWACS  -  767 
AWACS  RSIP 

AWACS  Comm  Extend  Sentry 
AWACS  Engine  Extent  Sentry 
AWACS  PDM  Re-Engineering 
Avionics  Strategic  Planning 
B-1B  Aircraft  Program  Director 
B-1B  Aircraft  Conv  Munition 
B-1B  Computer  System  Upgrade 
B-1B  Defensive  System  Upgrade 
B-1B  Mission  Planning  System 
B-1B  Training  System 
B-2  Aircraft  Training  System 
B-2  Aircraft  Integration 
B-2  Aircraft  Production 
B-2  Engine 
B-2  MSN  Planning 
Base  Information  Protection 
Big  Safari 

Broad  Based  Environment  for  Test 
C-17  ATS  Program  Branch 
CV-22 
C-32  (VC-X) 

Cl 30  H2  Aircrew  Training  System 
C-130J  Aircraft 

C-141  Aircrew  Training  System 
CCAWS  Project  Office 
Coatings  Technology  Integration 
Cockpit  Air  Bag  System 


Combat  Intelligence  System  (CIS) 
Command  and  Control  SPO 
Command  Center  Processing 
Common  Mapping  Production 
System  Common  Missile  Warning 
Directed  Infrared  Countermeasures 
DoD  Advanced  Automation  Systems 
DSU-33 

Embedded  GPS/INS 
Engine  Model  Derivative  Program 
Engine  Component  Improvement 
F-16 

F-16  Block  30  Targeting  Pod 
F-16  Common  Configuration  Implem 
F-16  Common  Missile  Warning 
F-16FMS 

F-16  Night  Vision  Imaging  System 

F-16RWR 

F-22 

F-22  Aircraft  Air  Vehicle 
F-22  Aircraft  --  F-199  Engine 
F-117A  Aircraft 

FI  17  Engine  -  Propulsion  for  C-l 
FI  17-PW-100  Engine  -  Propulsion 
Ground  Theater  Air  Control  System 
H-l  Marine  Upgrades  Program 
HAE  UAV 
HIDAR  &  MEECN 
IDECM 

Intelligence  and  Information  War 
Intelligence  Data  Handling  System 
IDHS  -  Automated  Message 
Handling 

IDHS  -  MAXI  Integrated  Avionics 
IDHS  -  Sentinel  II  Integration 
IDHS  -  IC4I  Intelligence  Receivers 
ITAS 

Joint  Advanced  Strike  Technology 
Joint  Direct  Attack  Munition 
Joint  Helmet  Mounted  Cueing  Sys 
Joint  Primary  Aircraft  Training 
Joint  Programmable  Fuze  (JPF) 

Joint  Service  Electronic  Combat 
Joint  STARS 

Joint  STARS  -  Deployable  Mission 
Joint  STARS  -  Flight  Crew  Train  Sys 
Joint  STARS -TADIL-J 
JTIDS 


KC-135  Aircrew  Training  System 
KC-135  Improved  Aerial  Refueling 
Kiowa  Warrior 
Kiowa  Warrior 

Large  Aircraft  Directed  Infrared 
Longbow  Apache 
MC-130H  Combat  Talon  II 
MILSTAR  Terminal  Command  Post 
Pr  Mission  Data  Preparation  System 
Mission  Planning  Systems  (MPS) 
Multiple  Launch  Rocket  System 
MLRS  Project  Office  -  IFCS. 

MLRS  Project  Office  -  ILMS 
Mod  Miniature  Receive  Terminal 
Non-Developmental  Airlift  Aircraft 
Non-Line  of  Sight  Project  Office 
Pacer  Speak 
Patriot  Project  Manager 
Peace  Fenghuang,  AFMSS 
Peace  Fox  VI,  AFMSS 
Peace  Sun  IX,  AFMSS 
Propulsion  Development  Systems 
Quiet  Knight  (SOF) 

RAH  -  66  Commanche  -  T-800  Eng 
RAH  -  66  Commanche  Program 
RAH  -  66  Commanche  Airframe 
R/IGS  -  Eagle  Vision/Commercial 
RFMETS 

Space  Surveillance  Network  Improve 
SOFPARS 

SR-71  Joint  Airborne  Sigint  Sys 
SR-71  Reactivation 
STINGER  Product  Office 
Surveillance  and  Control  SPO 
T-2S  Simulator  For  Electronic  Co 
T-3A  Enhanced  Flight  Screener 
TADS/PNVS 
TACMS-BAT  Army 
THAAD  Radar  Product  Manager 
Theater  Deployable  Communication 
Theater  Missile  Defense 
Tri-Service  Standoff  Attack  Missile 
Utility  Helicopters  (Black  Hawk) 
Voice  Processing  Training  System 
Voice  Comm  Switch  System 
Wind  Corrected  Munitions  Dispenser 


Chapter  7 


Methods  Used  to  Analyze  and  Present  the 
Survey  Results _ _ 


The  method  used  to  analyze  any  survey  results  depends  greatly  on  the  type  and  form  of 
data  collected.  The  three  surveys  entailed  in  this  research  were  used  to  gather  both  factual  and 
subjective  data,  in  continuous,  discrete,  and  categorical  formats.  Most  of  the  analysis  was 
accomplished  using  standard,  well-documented  statistical  techniques,  such  as  independent 
samples  T-test,  and  paired  samples  T-test.  These  methods  will  not  be  described  here,  but  the 
reader  can  refer  to  any  number  of  texts  that  cover  on  standard  statistical  procedures.86  The 
methods  used  to  analyze  the  factors  involved  in  developing  initial  project  schedules  will  be 
discussed  in  detail. 


86  George  Snedecor  and  William  Cochran  Statistical  Methods  8th  Edition  Iowa  State  University  Press  Ames  Iowa 
1989. 
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This  chapter  will  also  discuss  the  various  methods  used  to  present  the  survey  results  and 
the  reasons  for  their  use.  It  will  identify  the  benefits  and  shortcomings  of  each  method  so  the 
reader  fully  understands  what  is  and  is  not  presented. 


A.  Types  of  Data  Collected  and  Types  of  Questions  and  Scales  Used 

Both  factual  and  opinion  data  were  collected  through  the  surveys.  Factual  data  were 
collected  using  both  continuous  and  categorical  responses.  Opinion  data  were  collected  using 
dichotomous,  rank  ordering,  and  interval  scales. 

A. l  Factual  Data 

Where  possible,  factual  data  were  collected  using  continuous  variables.  Examples 
include  questions  about  budgets,  schedules,  and  various  other  factors  reported  as  continuous 
variables  such  as  dollars,  months,  or  number  of  people.  These  responses  were  used  as 
continuous  variables  throughout  the  analysis.  Other  types  of  factual  data  were  collected  using 
categories  or  dichotomous  responses.  These  included  yes/no  questions  and  selections  among  a 
range  of  choices. 

And  example  of  a  continuous  variable  question  is: 

B. l  What  is  this  project’s  current  estimate  or  actual  budget  expenditure,  and  what  were  the 
initial  planned  expenditures  for  all  program  expenses  in  each  of  the  following  program  phases? 
Use  actual  or  projected  then  year  dollars. 

CURRENT  ESTIMATE  INITIAL  PLAN 

Pre-EMD  ($  million)  ($  million) 

EMD  ($  million)  ($  million) 

Production  ($  million)  ($  million) 

An  example  of  a  dichotomous  question  is: 

C. 9  Did  the  Government,  through  its  RFP  or  other  means,  specify  an  expected  project 

schedule  to  the  contractors? 

□  No. 


□  Yes. 
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Finally,  an  example  of  a  question  with  a  discrete  set  of  choices  is: 

A.5  Which  of  the  following  categories  best  describes  the  type  of  system  this  project  is  developing? 

□ 

□ 

□ 

□ 

These  questions  could  be  answered  based  on  facts  available  to  the  respondent. 

A.2  Opinion  Data 

Not  all  the  necessary  information  could  be  obtained  from  factual  data.  The  other  types  of 
data  desired  included  statements  of  respondents’  opinions,  beliefs,  and  views  on  various  aspects 
of  a  development  project.  Since  the  respondents  were  the  individuals  making  most  decisions  and 
thus  having  the  greatest  impact  on  the  aspects  being  measured,  their  views  and  opinions  were 
significant.  The  actions  they  took  and  the  decisions  they  made  were  based  on  their  perception  of 
events  and  the  information  they  were  provided.  What  the  surveys  measured  was  their 
interpretation  of  the  information  on  which  they  base  their  decisions. 

These  responses  were  collected  in  several  formats,  including  choices  between  two 
opposing  responses,  choices  among  preset  categories,  rank  orders  of  a  number  of  items,  and 
information  on  an  ordered  scale.  Examples  are  provided  for  each  type  of  question  below. 

Dichotomous  Answers 

When  it  was  deemed  appropriate  to  force  respondents  to  choose  between  two  opposing 
answers,  yes/no  or  dichotomous  questions  were  used.  These  questions  asked  respondents  to 
select  one  answer  best  representing  their  project  from  two  opposing  statements.  The  answers  are 
presented  as  percentage  of  responses,  or  they  are  used  to  separate  groups  of  cases  for  testing  of 
other  variables.  An  example  of  dichotomous  questions  is: 


Aircraft  (airframe  and  mechanical  systems)  □  Electronic  system  (non-aircraft) 

Aircraft  (avionics  and  electronic  systems)  □  Missile  or  munitions 

Aircraft  (propulsion)  □  Software-dominated  system 

Spacecraft  or  launch  system  □  Other: _ _ _ 
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A.16  From  each  of  the  following  pairs  of  statements,  please  select  the  one  from  each  row 
which  best  represents  this  project: 

Statement  A  or  Statement  B 


□  This  project  was  started  as  the  result  □  This  project  was  started  at  a  senior 

of  the  normal  service  planning  process.  leader’s  direction. 


□  This  project  is  intended  to  meet  a  □  This  project  in  intended  to  meet  a  future 

current  operational  deficiency.  or  projected  operational  deficiency 


Selection  of  Categories 

Where  information  could  be  categorized  adequately,  respondents  were  asked  to  select 
among  different  responses  based  on  specific  word  descriptions.  The  resulting  data  were  used  to 
sort  the  projects  into  different  groups  to  check  for  differences  in  other  variables.  An  example  is: 


A.  12  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
product  technologies  required  by  this  project,  relative  to  existing  systems: 

□  Revolutionary  new  core  technologies  or  concept. 

□  New  generation  of  product  architecture  or  platform. 

□  Incremental  improvements  to  an  existing  generation  of  product  architecture  or  platform. 

□  Little  or  no  change  to  the  existing  product  technology. 


Rank-Ordering 

In  some  cases  respondents  were  asked  to  rank-order  a  list  of  factors  that  may  have 
influenced  their  decisions.  This  method  was  used  when  the  order  of  items  was  important  and  they 
were  few,  usually  four  or  less.  These  data  were  used  to  present  the  relative  ranking  of  project 
objectives  in  raw  format,  and  to  categorize  the  projects  for  analysis  of  other  variables.  A  sample 
of  this  type  of  rank  question  is: 
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A.13  Please  rank  the  order  of  importance  of  the  following  considerations  in  meeting  this 
project’s  objectives  (“1”  indicates  the  item  with  the  greatest  importance  in  meeting  this  project’s 
objectives,  “4”  indicates  the  item  with  the  least  importance  in  meeting  this  project’s  objectives): 

_ Low  acquisition  cost. 

_ Low  operational  and  support  cost. 

_ _  Short  schedule  to  reach  operational  capability. 

_  Superior  technical  performance. 


Interval  Scales 

For  questions  where  the  strict  order  was  not  important,  or  where  the  number  of  item 
made  rankings  difficult  and  time  consuming,  a  seven-point  interval  scale  was  used  to  collect 
information.  The  seven-point  scale  was  seen  as  providing  sufficient  distinction  among  factors  but 

not  overloading  respondents  with  choices.  This  scale,  while  discrete  in  nature,  represented  a 

* 

continuous  and  equal  interval  range  of  the  impact  of  various  factors,  with  one  typically 
representing  the  least  impact  and  seven  representing  the  most  impact.  An  example  of  this  type  of 
question  is: 


C.6  Please  indicate  the  extent  to  which  you  believe  the  following  factors  helped  determine  the 


length  of  the  government’s  initial  schedule  for  this  project: 

No 

Some 

Defining 

Impact 

Impact 

Impact 

User’s  Desired  Schedule 

i 

2 

3 

4 

5 

6 

7 

Service  Leadership  Desired  Schedule 

i 

2 

3 

4 

5 

6 

7 

Service  Planning  Process 

1 

2 

3 

4 

5 

6 

7 

Expected  Development  Funding  Availability  1 

2 

3 

4 

5 

6 

7 

Expected  Production  Funding  Availability 

i 

2 

3 

4 

5 

6 

7 

Technology  Development 

i 

2 

3 

4 

5 

6 

7 

Engineering  Requirements 

i 

2 

3 

4 

5 

6 

7 

Manufacturing  Process  Development 

i 

2 

3 

4 

5 

6 

7 

Testing  Requirements 

i 

2 

3 

4 

5 

6 

7 

Support  Requirements 

i 

2 

3 

4 

5 

6 

7 

Dependence  on  another  program 

i 

2 

3 

4 

5 

6 

7 

Other: 

i 

2 

3 

4 

5 

6 

7 
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The  information  collected  from  these  questions  was  analyzed  to  determine  the  relative 
importance  of  various  factors  influencing  initial  schedule  development  and  other  areas  of 
interest.  The  method  used  to  analyze  the  responses  was  employed  extensively  in  this  thesis  and 
will  be  discussed  in  more  detail. 

During  analysis,  data  from  the  three  surveys  was  pooled  where  appropriate  and  where 
there  was  no  significant  differences  among  responses.  Where  significant  differences  were  noted 
among  the  three  surveys,  the  analysis  discussed  those  differences  in  terms  of  particular  questions. 
In  general,  the  surveys  revealed  marked  agreement  among  responses  to  the  three  surveys. 

What’s  more,  no  significant  differences  appeared  among  surveys  completed  for  the  same  project 
at  the  Pentagon,  Program  Office,  or  contractor  levels.  However,  the  number  of  overlapping 
surveys  for  a  given  project  was  relatively  small  (8),  making  analysis  of  matched  surveys 
statistically  unsound.  Thus  no  matching  of  projects  among  the  Pentagon,  Program  Office,  or 
contractor  surveys  was  used  in  the  analysis. 


B.  Methods  Used  to  Analyze  the  Interval-Scale  Data  and  to  Determine 
the  Relative  Impact  of  Various  Factors 

Several  methods  were  used  to  analyze  the  survey  results,  including  visual  comparison  of 
raw  aggregated  data,  comparison  of  the  standard  errors  of  the  means,  and  paired  samples  tests. 

The  objective  was  to  distinguish  the  influence  of  different  factors  on  project  schedules. 

Each  respondent  rated  the  various  factors  against  the  same  scale.  A  difference  in  the  rating 
of  two  factors  by  that  respondent  indicated  a  difference  in  the  perceived  impact.  If  a  factor  was 
systematically  reported  to  have  a  larger  impact  than  another,  this  was  interpreted  to  mean  that  the 
factor  has  a  larger  impact  on  project  schedules  across  the  development  system. 

When  combining  individual  project  responses  to  provide  a  description  of  the  overall 
schedule  development  process,  one  can  look  at  the  distribution  of  the  responses  to  each  question 
separately,  or  at  the  differences  among  ratings  of  each  factor  within  each  survey.  Various  statistical 
tests  were  used  to  determine  if  the  differences  among  responses  for  factors  were  statistically 
significant.  Where  statistical  differences  existed  among  the  distribution  of  responses  for  various 
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factors,  and  the  distribution  of  responses  for  each  possible  pair  of  factors,  these  results  were 
interpreted  to  mean  there  was  a  difference  across  the  range  of  projects  included  in  the  analysis.  The 
null  hypothesis  was  that  there  is  no  difference  among  various  responses  for  the  two  factors,  which 
would  indicate  there  is  no  difference  between  the  impact  of  the  factors.  The  alternative  hypothesis 
was  that  responses  to  the  various  factors  were  different,  indicating  a  difference  in  the  impact  on 
initial  schedules  between  the  two  factors  across  survey  respondents.  Various  statistical  tests  were 
used  to  show  that  there  was  or  was  not  a  statistical  difference  between  the  various  factors.  For  pairs 
of  factors  between  which  statistical  methods  cannot  distinguish ,  those  factors  were  interpreted  to 
have  an  equal  impact  on  the  schedule  development  process. 

B.l.  Visual  Inspection  of  Raw  Aggregated  Data  Distributions 

Before  the  statistical  analysis  was  conducted,  the  database  was  visually  inspected  for  the 
distribution  of  responses  to  each  question.  The  visual  inspections  indicated  that  several  of  the 
responses  had  bi-modal  distributions,  with  spikes  at  no  impact  and  some  impact.  In  many  cases 
the  data  collected  did  not  appear  to  follow  a  normal  distribution  somewhat  limiting  the  number 
of  statistical  tests  that  could  be  used. 

Standard  statistical  measures  were  used  during  the  initial  screening  of  the  data.  Below 
are  samples  of  the  tabular  and  graphical  data  formats  used  to  inspect  the  data 


164  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  DevelopmentProcess 


Frequency  of  Responses 

No 

Some  Defining 

Impact 

Impact  Impact 

Order 

Description 

1  2 

3  4  5  6  7 

Asked 

1 

Users  Desired  Schedule 

13 

7 

14 

41 

28 

43 

34 

4 

Expected  Development  Funding 

26 

6 

13 

31 

38 

37 

28 

5 

Expected  Production  Funding 

33 

8 

11 

32 

35 

27 

27 

7 

Engineering  Requirements 

15 

18 

35 

46 

28 

24 

12 

9 

Testing  Requirements 

22 

22 

31 

42 

29 

25 

9 

2 

Leadership  Desired  Schedule 

35 

15 

27 

31 

30 

22 

17 

6 

Technology  Development 

32 

22 

30 

40 

21 

25 

8 

3 

Force  Planning 

39 

17 

29 

30 

26 

22 

11 

11 

Dependence  on  Another  program 

53 

30 

14 

19 

18 

18 

23 

10 

Support  Requirements 

35 

45 

41 

27 

17 

10 

1 

8 

Manufacturing  Development 

46 

36 

30 

28 

13 

19 

4 

Table  7-1:  Distribution  of  Response  to  the  Effect  of  Various  Factors  on  the  Length  of  the 
Initial  Project  Schedule  (Sorted  by  Median  Value) 


Med  Median*  Mode  Sum 
ian 


N  Quartile 
Range 

25%  50%  75% 


Mean 


1  User's  Desired  Schedule 

5 

5.028 

6 

869 

180 

4 

5 

6 

4.828 

4  Expected  Development  Funding 

5 

4.841 

5 

809 

179 

3 

5 

6 

4.520 

5  Expected  Production  Funding 

5 

4.552 

5 

736 

173 

3 

5 

6 

4.254 

7  Engineering  Requirements 

4 

3.951 

4 

708 

178 

3 

4 

5 

3.978 

2  Leadership  Desired  Schedule 

4 

3.862 

1 

671 

177 

2 

4 

5 

3.791 

9  Testing  Requirements 

4 

3.836 

4 

685 

3 

4 

5 

3.806 

6  Technology  Development 

4 

3.571 

4 

637 

178 

2 

4 

5 

3.579 

3  Force  Planning 

4 

3.559 

1 

619 

174 

2 

4 

5 

3.557 

1 1  Dependence  on  Another  Program 

3 

2.886 

1  . 

590 

175 

1 

3 

5 

3.371 

8  Manufacturing  Development 

3 

2.727 

1 

527 

176 

1 

3 

4 

2.994 

10  Support  Reauirements 

3 

2.709 

2 

508 

176 

2 

3 

4 

2.886 

*  Calculated  as  if  from  grouped 

data 


Table  7-2:  Distribution  Statistics  for  Responses  to  the  Effect  of  Various  Factors  on  the 
Length  of  the  Initial  Project  Schedule 


Number  of  Projects 
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Impact  of  Engineering  Requirements 
on  the  Length  of  the  Initial  Project  Schedule 


No  Some  Defining 

Impact  Imnact  Tmnact 


1  2  3  4  5  6  7 


Engineering  Requirements 


Figure  7-1:  Distribution  of  Responses  for  Impact  of  Expected  Development  Funding  and 
Engineering  Requirements  on  the  Length  of  the  Initial  Project  Schedule 
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One  then  can  compare  the  means  and  the  standard  deviations  to  determine  which  factors 
were  rated  higher  than  the  others.  However,  comparisons  based  on  the  mean  and  standard 
deviations  are  flawed,  as  they  assume  normal  or  near-normal  distributions,  which  were  not 
apparent  in  some  cases.  What’s  more,  comparison  of  the  means,  while  illustrative  and  helpful, 
may  not  reveal  significant  differences  that  may  exist  in  the  data,  as  such  comparisons  are  less 
efficient  from  a  statistical  standpoint  than  other  methods  available.  Comparison  of  the  standard 
error  of  the  mean  and  various  paired  sample  methods  allows  the  survey  data  to  be  used  on  a 
project-by-project  basis,  as  opposed  to  aggregate  data  only. 


B.2.  Comparison  of  the  Sample  Means  for  Different  Factors 

The  mean  of  the  responses  to  a  given  question  provided  a  view  of  the  average  response  of 
those  surveyed  across  the  projects.  This  mean  estimated  the  average  of  the  entire  population 
within  the  bounds  of  standard  error.  Averaging  across  a  wide  array  of  projects  with  different 
characteristics  may  hide  important  variations  among  those  projects.  Since  the  objective  is  to 
describe  the  system  as  a  whole,  the  responses  averages  have  some  utility.  However,  they  do  not 
indicate  how  the  various  factors  affected  an  individual  project. 

During  the  analysis,  the  survey  responses  were  subdivided  into  an  array  of  categories, 
with  the  answers  to  various  questions  used  to  look  for  systematic  differences  among  the 
categories.  Those  included  the  cost  of  a  project,  the  type  of  project,  the  type  of  contract,  the 
development  phase  of  the  project,  and  the  amount  of  technological  advance.  The  rank-order  of 
the  average  responses  to  most  questions  was  found  to  be  consistent  or  nearly  consistent 
irrespective  of  the  category  or  type  of  project.  The  analysis  of  the  subsets  of  projects  based  on 
their  characteristics  is  discussed  in  the  analysis  of  individual  questions. 

To  analyze  the  data,  the  mean  responses  and  the  standard  error  of  the  mean  were 
determined  using  the  survey  results.  Even  if  the  distribution  of  the  raw  responses  to  a  question 
were  not  normal,  the  large  number  of  projects  obtained  through  the  surveys  indicated  that  the 
sampling  distribution  of  the  mean  will  be  near  normal  in  shape,  and  will  allow  the  use  of  the 
standard  T  test  to  compare  the  distribution  of  the  sample  mean  of  responses. 
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The  sample  mean,  and  the  sample  standard  deviation  and  standard  error  of  the  mean,  are 
calculated  with  the  equations  below. 

Sample  Mean  =  X  =  X  (X/N)  Equation  7-1 

Where 

N  is  the  number  of  survey  responses 

Xj  is  the  response  for  each  individual  project/survey 

2  1/2 

Sample  Standard  Deviation  =  S  =  (X  (X,  -  X)  /(N-l))  Equation  7-2 

1/2 

Standard  Error  of  the  Mean  =  Sx_  =  S/(N)  Equation  7-3 


The  results  for  the  example  used  earlier  are: 


Expected  Development  Funding 

X  =  4.52 
S  =  1.93 
N=  179 

Standard  Error  of  the  Mean  =  0. 144 


Engineering  Requirements 

X  =  3.98 
S  =  1.64 
N=  178 

Standard  Error  of  the  Mean  =  0.123 


Plotting  the  means  of  the  responses— and  the  standard  error  of  the  means— for  the  impact 
of  various  factors  on  initial  development  schedule  allows  the  average  of  the  responses  and  the 
expected  errors  to  be  viewed  graphically. 
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Deydcpmait  of  thelnitid  Reject  SchedJe 

Fetters  Meeting  the  Lai^hcf  the  Iritid  Prqect  SchedJe 

User’s  Desires 
Expected  Development  Funding 
Expected  Prod  Funding 
Engineering  Requirements 
Testing  Requirements 
Service  Leadership  Desires 
Technology  Development 
Service  Planning 
Dependence  on  Another  Program 
Manufacturing  Process  Development 
Support  Requirements 

1  2  3  4  5  6  7 

No  Impact  Some  Impact  Defining  Impact 

W&T\+/-  I  St 

Figure  7-2:  The  Average  and  Standard  Error  of  the  Mean  for  Responses  to  the  Impact  of 
Various  Factors  on  the  Length  Initial  Project  Schedule 

The  unpaired  sample  t  test  can  be  used  to  compare  the  mean  of  two  factors.  The  null 
hypothesis  is  that  there  is  no  difference  between  the  mean  responses  to  the  impact  of  two 
scheduling  factors.  The  following  equation  is  used  to  calculate  the  t  value  for  a  comparison  of 
the  means  of  two  independent  samples  tests. 

t'  =  (X,  -  XjWSjVN,  +  S22/N2)w  Equation  7-4" 

where:  X,  is  the  mean  of  the  responses  for  the  first  factor 

X2  is  the  mean  of  the  responses  for  the  second  factor 

S,  is  the  standard  deviation  for  the  responses  to  the  first  factor 
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S2  is  the  standard  deviation  for  the  responses  to  the  second  factor 
N,  and  N2  are  the  number  of  projects  surveyed 

The  degrees  of  freedom  for  the  test  are  approximated  to  allow  the  ordinary  t  distribution 
tables  to  be  used.  The  approximation  for  the  number  of  degrees  of  freedom  is  given  by: 

v'  =  (\),+  0)2)  7(a),2  /v,  +0)22  /  V2)  Equation  7*5" 

where:  u,  =  S,2/n, 

u2  =  S22/n2 
v,=  n,-l 
v2=  n2-l 

The  approximate  degrees  of  freedom  for  t’  are  given  by  v’,  which  is  rounded  down  to  the 
nearest  integer  so  that  standard  t  tables  can  be  used." 

Using  the  example  discussed: 

t’=  (4.52  -  3.98)/((1.932/179)+(1.642/178)) m  =  2.849 

v’=  ((1.932/179)+(1.642/178))2/  ((1.932/179)2/(178)+  (1.642/178)2/(177))  =  346.58 
Approximate  Degrees  of  Freedom  =  346 

The  null  hypothesis— that  there  is  no  difference  between  the  two  samples— can  be  rejected  based 
on  the  value  of  t’  and  the  number  of  degrees  of  freedom  at  difference  levels  of  confidence.  With 
the  sufficiently  large  sample  sizes  captured  in  the  research,  t’  can  be  estimated  by  t,  and  either 
calculated  from  the  normal  deviate  or  looked  up  in  any  standard  statistics  reference  book.  The  5 
percent  and  the  0.5  percent  level  of  confidence  for  t’  is: 

t’005=  5%  confidence  level  =  1.96  (346  df) 

fo.^  =  0.5%  confidence  level  =  2.807  (346  df)90 
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Using  the  t  distribution,  the  probability  that  the  average  of  the  two  samples  will  be  equal 
is  0.5  percent— well  below  the  5  percent  typically  required  for  demonstrating  statistical 
significance.  More  powerful  statistical  methods  are  available  that  use  the  survey  information  on  a 
project-by-project  basis. 

B.3.  Paired-Samples  Statistical  Methods 

Analyzing  the  multiple-response  questions  using  paired-sample  statistical  methods  allows 
the  relative  importance  of  factors  to  be  compared  on  a  project-by-project  basis.  The  distribution  of 
responses  can  then  be  evaluated  to  determine  if  there  is  a  statistical  difference  between  them.  The 
pairing  of  the  data  within  a  single  project  ensures  that  each  factor  is  affected  by  the  same  events 
and  interpreted  by  the  same  person.  Using  the  information  in  this  manner  removes  the  effects  of 
non-related  factors,  as  each  project  response  is  compared  only  with  responses  from  the  same 
project.  By  using  the  paired  responses  from  each  project  separately,  it  is  possible  to  eliminate 
differences  that  result  from  the  specific  nature  of  the  projects,  or  from  respondents’  different 
interpretations  of  the  scale. 

When  comparing  two  factors,  the  response  for  one  factor  is  subtracted  from  the  response  for  the 
other  factor.  If  there  is  no  systematic  difference  between  the  two  factors,  the  distribution  difference  of 
the  paired  responses  should  be  centered  on  no  difference.  Continuing  with  the  earlier  example,  the 
reported  differences  between  the  impact  of  the  expected  development  funding  and  the  engineering 
requirements  is  shown  in  Figure  7-5.  A  negative  number  indicates  that  the  engineering  requirements 
were  rated  as  having  a  larger  impact  on  the  length  of  the  initial  project  schedule,  and  a  positive  response 
indicates  that  expected  development  funding  had  a  greater  impact  on  that  project.  A  zero  response 
indicates  that  the  respondent  rated  them  as  equally  important.  The  numbers  on  the  horizontal  axis 
indicate  the  difference  between  the  two  responses  by  a  single  respondent.  The  numbers  on  the  vertical 
axis  represent  the  number  of  respondents  reporting  the  same  difference  between  the  two  responses. 


89  Snedecor  and  Cochran.  Statistical  Methods.  Analysis  of  Independent  Samples  when  o,*a2  Pg.  97 
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The  data  are  interpreted  based  on  the  assumption  that  the  larger  the  difference  between  the 
reported  responses,  the  larger  the  differences  in  the  impact  of  the  two  factors.  For  example,  if  a 
respondent  reported  that  the  engineering  requirements  had  no  impact  (and  thus  rated  them  a  one)  and 
chose  funding  availability  as  the  defining  impact  (thus  rating  it  a  seven),  the  difference  between  the  two 
responses  would  be  6.  This  would  indicate  a  larger  difference  than  in  the  case  of  someone  who  awarded 
the  engineering  requirements  a  5  and  the  funding  availability  a  6,  as  the  difference  between  those  two 
values  would  be  1.  If  the  factors  are  equally  important  or  there  are  no  systematic  differences  between 
them,  there  should  be  an  equal  number  of  responses  indicating  a  positive  and  negative  difference.  The 
existence  of  such  a  difference  can  be  tested  using  various  paired-sample  statistical  methods. 
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Figure  7-3:  Distribution  of  Differences  between  the  Reported  Impact  of  Development 
Funding  and  Engineering  Requirements  on  the  Length  of  the  Initial  Project  Schedule 


Three  primary  paired-comparison  methods  can  be  used  with  the  type  of  scale  data 
collected  by  the  surveys.  These  include  paired-samples  t-test  corrected  for  continuity,  the  non- 
parametric  sign,  and  Wilcoxon  signed-rank  tests.  Parametric  tests  make  certain  assumptions 
about  the  distributions  of  the  data,  such  as  that  it  is  continuous  and  normally  distributed.  This 
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could  not  be  assured  for  this  analysis,  so  the  parametric  tests  were  not  used.”  Non-parametric 
tests  do  not  require  such  assumptions.  Parametric  tests  are  usually  more  powerful  than  the  non- 
parametric  tests,  as  they  are  less  likely  to  indicate  a  significant  difference  when  there  is  not  one, 
and  they  are  less  likely  to  indicate  that  there  is  no  significant  difference  when  there  is  one. 

Sign  Test 

The  sign  test  is  a  relatively  simple  test  that  looks  solely  at  the  direction  of  the 
differences  between  two  matched  responses.  It  does  not  measure  the  magnitude  of  the 
difference.  If  there  is  no  systematic  difference  in  the  responses  for  the  two  factors,  there  should 
be  roughly  an  equal  number  of  cases  where  one  factor  is  rated  higher  than  the  other.  The  null 
hypothesis  is  that  the  two  distributions  are  equal,  and  that  each  paired  difference  has  an  equal 
chance  of  being  positive  or  negative.  A  large  disparity  in  the  number  of  positive  and  negative 
differences  would  indicate  that  the  distributions  are  not  equal,  and  that  the  null  hypothesis  can  be 
rejected  for  the  two  responses.  This  test  is  accomplished  for  every  possible  pair  of  responses  (the 
test  ignores  ties.  The  results  of  the  sign  test  for  expected  development  funding  and  engineering 
requirements  is  provided  below  as  an  example. 

In  88  cases,  responses  rated  expected  development  funding  as  having  more  impact  than 
engineering  requirements  on  the  length  of  initial  project  schedule.  The  engineering  requirements 
were  rated  higher  in  57  cases,  and  in  33  cases  the  factors  were  rated  equally. 


Example  Sign  Test 

Expected  Development  Funding  Rated  Larger  88  +  Diffs 

Engineering  Requirements  Rated  Larger  57  -  Diffs 

Rated  as  Equal  33 — Ties 

Number  of  Cases  178 

Number  of  Non-Tied  Cases  145 


91  The  results  using  the  parametric  and  non-parametric  tests  were  similar  where  checked. 
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The  normal  deviate  corrected  for  continuity  is  calculated  by: 

Z  =  (I2r-nl-l)/n  Equation  7-6” 

Where  r  is  the  number  of  positive  or  negative  values  and  n  is  equal  to  the  number  of  non-tied 
samples.  For  the  example: 

Zc  =  (I2*88-145I-1)/145,/2  =  2.4914 

The  two-tailed  significance  can  be  calculated  from  the  cumulative  normal  frequency  distribution, 
or  it  can  be  obtained— as  it  was  in  this  case— from  a  statistics  table  for  normal  distribution.”  The 
probability  that  the  two  factors  are  actually  equal  and  that  the  result  is  a  statistical  fluke  is 
0.0180. 


2-Tailed  P=  0.0180 

The  sign  test  is  not  as  efficient  in  determining  differences  between  populations  as  tests 
that  use  the  magnitude  of  the  difference  between  responses.  The  other  non-parametric  test,  the 
Wilcoxon  signed-rank  test,  uses  the  magnitude  as  well  as  the  sign  of  the  difference,  and  so  is 
considered  more  decisive. 


Wilcoxon  Signed-Rank  Test 

Because  the  Wilcoxon  signed-rank  test  is  a  non-parametric  test,  it  does  not  require  the 
assumption  of  a  normal  distribution.  The  use  of  this  test  assumes  that  the  larger  the  difference  in 
ratings  between  two  factors,  the  larger  the  differences  in  the  factors’  effects. 

In  the  Wilcoxon  signed-rank  test,  the  absolute  values  of  the  differences  between  the 
matched  pairs  of  data  are  rank-ordered  from  1  to  n,  with  the  smallest  difference  as  1.  For  the 


92  George  Snedecor  and  William  Cochran  _Statistical  Methods_  8th  Edition  Iowa  State  University  Press  Ames  Iowa 
1989.  Pg.  139 

93  George  Snedecor  and  William  Cochran  _Statistical  Methods_  8th  Edition  Iowa  State  University  Press  Ames  Iowa 
1989.  Pg  464. 
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responses  with  equal  differences,  the  average  of  the  ranks  is  used.  The  rankings  associated  with 
the  positive  differences  and  the  rankings  associated  with  the  negative  differences  are  then 
summed  separately.  Responses  with  no  difference  between  the  factors  are  not  included.  The 
smaller  of  the  two  sums  is  then  used  to  calculate  the  probability.  For  large  numbers  of  pairs  the 
normal  deviate,  corrected  for  continuity,  is  given  by: 

Zc  =  (|X-  T  -  0.5)/ct  Equation  1-T 

Where  a  =((2n+l)  p/6)1/2 

|X=n(n+l)/4 

T  =  smaller  of  the  sum  of  the  positive  and  negative  ranks 
n  =  number  of  pairs 

The  probability  that  there  is  no  difference  between  the  two  factors  can  be  determined 
using  the  two-tailed  significance  obtained  from  the  cumulative  normal  frequency  distribution. 
For  a  normal  distribution,  the  null  hypothesis-that  the  two  factors  are  rated  equal-can  be 
rejected  at  the  0.05  level.  Zc>  1.96  equates  to  a  probability  of  0.05,  which  signifies  rejection  of 
the  null  hypothesis  and  indicates  that  the  two  factors  are  actually  rated  differently. 

An  example  using  the  Wilcoxon  signed-rank  T  test  to  measure  the  impact  of  expected 
development  funding  and  engineering  requirements  is  shown  below.  The  table  on  the  next  page 
shows  the  responses  ordered  by  the  absolute  value  of  the  differences  between  the  two  factors.  A 
positive  value  indicates  that  expected  development  funding  was  rated  as  having  a  larger  impact, 
and  a  negative  value  indicates  that  engineering  requirements  were  rated  as  having  a  larger 
impact.  The  responses  are  then  ranked  from  1  to  145  by  the  absolute  value  of  the  differences. 
The  adjusted  rankings  account  for  responses  with  equal  differences  by  taking  the  average  of  the 
rank  of  the  responses  and  as  illustrated  in  Table  7-3  below. 


94  George  Snedecor  and  William  Cochran  _Statistical  Methods.  8th  Edition  Iowa  State  University  Press  Ames  Iowa 
1989.  Pg  141. 
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Table  7-3:  Ranking  Used  for  Wilcoxon  Signed-Rank  Test  on  the  Impact  of  Expected 
Development  Funding  and  Engineering  Requirements  on  Length  of  Initial  Project 
Schedule  (33  Responses  with  No  difference  Are  Not  Included) 
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The  sum  of  the  ranking  for  responses  rating  expected  development  funding  higher  is 
6737,  and  the  sum  of  the  ranking  for  those  that  rate  engineering  requirements  higher  is  3916. 

Using  the  minimum  of  the  two  sums  as  specified  by  the  Wilcoxon  method, 

n  =  number  of  pairs  =  145 

jx=  n(n+l)/4  =  145(145+l)/4  =  5292.5 

o  =((2n+l)  p/6) 1/2  =  ((2*145+l)*5292.5/6) 1/2  =  506.64 

T  =  smaller  of  the  sum  of  the  positive  and  negative  ranks  =  3916 


Zc  =  (p  -T-0.5)/a  =  (5292.5  -  3916  -  0.5)7506.64  =  2.7159 


The  probability  that  the  null  hypothesis  is  true,  and  that  there  is  no  difference  between  the  two 
factors,  is  determined  by  the  two-tailed  significance  obtained  from  the  cumulative  normal 
frequency  distribution.  For  example,  the  probability  that  the  two  factors  are  actually  equal  is 
0.0033. 


2-Tailed  P  =  0.0033 

This  is  significantly  beyond  the  0.05  level  typically  used  to  show  statistical  significance.  For  the 
analysis  used  in  this  survey,  the  statistical  tests  was  conducted  using  the  SPSS  statistical  data 
software  package  produced  by  SPSS,  Inc.” 

A  comparison  of  results  from  the  three  statistical  tests  showed  little  difference  among 
factors  when  they  were  approaching  statistically  significant  levels.  For  the  analysis,  the  signed- 
rank  test  was  selected  as  it  was  seen  as  the  most  efficient  test  that  did  not  rely  on  the  assumption 
of  a  normal  distribution  of  the  data,  which  could  not  be  guaranteed.  The  signed-rank  test  was 
used  for  analysis  except  were  specifically  noted. 
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B.4.  Development  of  Statistical  Groups  of  Factors 

The  paired-sample  statistics  tests  indicated  that  there  was  or  was  not  a  statistical 
difference  between  any  two  factors.  Because  there  were  typically  many  factors  rated  for  each  of 
many  questions,  presenting  the  results  as  a  factor-by-factor  comparison  for  each  pair  of  factors 
was  determined  not  to  be  impractical.  To  provide  a  method  for  presenting  and  discussing  the 
data  at  a  higher  level,  factors  that  could  be  distinguished  with  statistical  methods  were  separated 
into  different  statistical  groups,  and  factors  that  could  not  be  distinguished  from  each  other  were 
grouped  together.  The  resulting  groups  could  then  be  used  to  discuss  the  impact  of  the  different 
factors  at  a  higher  level. 

The  different  statistical  groups  were  determined  by  first  developing  a  factor  matrix 
indicating  the  probability  that  each  possible  pair  of  factors  were  equivalent.  This  matrix  was 
based  on  the  results  of  the  Wilcoxon  signed-rank  test,  and  its  initial  order  was  determined  by  the 
average  of  the  responses.  A  visual  inspection  of  each  matrix  looked  for  factors  that  appeared  out 
of  order  based  on  the  probabilities.  In  almost  all  cases,  ordering  the  factors  by  their  mean  values 
proved  adequate  to  determine  the  statistical  groups,  as  described  below. 

The  statistical  groups  were  determined  by  visually  inspecting  the  factor  matrix  and 
identifying  the  groups  of  factors  that  were  statistically  different  from  other  groups  of  factors. 
This  was  done  by  first  identifying  statistical  differences  between  adjacent  factors  within  the 
matrix,  and  then  checking  to  ensure  there  were  statistical  differences  with  other  non-adjacent 
factors.  A  minimum  of  85  percent  confidence  level  (P  =  0. 15)  was  used  to  indicate  statistical 
significance  between  adjacent  factors.  The  relatively  low  level  was  used  because  of  the  large 
number  of  factors  and  the  difficulty  of  separating  adjacent  factors.  Thus  appropriate  caution 
should  be  used  when  making  comparisons  between  adjacent  factors  across  the  statistical  groups. 
The  differences  between  non-adjacent  factors  across  statistical  groups  are  typically  at  much 
higher  levels  of  statistical  significance  than  the  0.15  level.  In  some  cases,  the  0.15  level  was  not 
met,  but  in  the  judgment  of  the  researcher  the  significant  difference  between  close  but  non- 
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adjacent  factors  indicated  that  the  factors  should  be  in  different  groups.  The  few  times  this 
occurred  a  dashed  line  signifies  that  the  adjacent  factors  are  not  statistically  different  but  that  the 
other  factors  are.  Generally,  adjacent  factors  within  the  same  statistical  grouping  cannot  be 
separated  with  confidence  based  on  the  data  obtained  through  the  surveys.  However,  factors  in 
one  statistical  grouping  can  be  separated  statistically  from  those  in  other  groupings. 

The  factor  matrix  and  the  statistical  groups  for  factors  affecting  the  length  of  a  project’s 
initial  schedule  is  shown  in  Figure  7-4  below. 


Users 

Desired 

Schedule 

Expected 

Develop 

Funding 

Expected 

Product 

Funding 

Enginee  Testing  Leader 
ring  Requirem  Desired 

Require  ents  Schedule 

ment 

Technol 

ogy 

Develop 

ment 

Force 

Planning 

Dependen  Manufa  Support 

ce  on  cturing  Require 

Another  Develo  ments 

Program  pment 

G 

r 

0 

u 

1 

4 

5 

7 

9 

2 

6 

3 

li 

8 

10 

p 

4.83 

4.52 

4.25 

3.98 

3.81 

3.79 

3.58 

3.56 

3.37 

2.99 

2.89 

Users  Desired  Schedule 

1 

0.23 

0,00 

ito# 

*  •* 

000 

tm 

' , 

m 

i 

Expected  Development  Funding 

0.23 

1 

0.15 

0.01 

flS! 

ill 

0.00 

IJ§ 

0.00 

0.00 

,0.00 

i 

Expected  Production  Funding 

::md 

0.15 

2 

0.12 

Ill 

'§M 

00 

0.00 

IS 

2 

Engineering  Requirements 

im 

0.12 

3 

0.17 

0.36 

IIP 

811 

M 

i® 

~3 

Testing  Requirements 

jp? 

ilfi 

0.17 

3 

0.90 

6.19 

0.39 

°:03 

0.00 

$1°° 

3 

Leadership  Desired  Schedule 

,  -SB •« 

0,00 

0.36 

0.90 

3 

0.18 

0.10 

0.03 

^St  s 

0.00 

3 

Technology  Development 

0,00 

o;ooi 

m 

m 

0.19 

0.18 

3 

0.87 

0.32 

0.00 

0.00| 

3 

Force  Planning 

0.00 

0.05 

0.39 

0.10 

0.87 

3 

0.25 

111 

i!g! 

3 

Depend  on  Another  Programs 

g§fg 

If 

0.03 

ms 

0.32 

0.25 

3 

0.11 

;  0.01 

3 

Manufacturing  Development 

0,00,  | 

ISi 

If0 

$0 

0.11 

4 

0.37 

“4 

Support  Requirements 

0.00  J 

,  '  ' 

11211 

000 

0.00 

0.00  , 

000 

0.00 

0.01 

0.37 

4 

4 

*  =  Significant  Difference  determined  by  Wilcoxon  Rank-Sign  Test 


Figure  7-4:  Illustration  of  Statistical  Grouping  Method  Used  in  Analyzing  Data  for  Factors 
Determining  Length  of  Initial  Project  Schedule 
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C.  Presentation  of  the  Data 

Throughout  the  thesis,  several  methods,  each  with  its  limitations,  are  used  to  present  the 
data.  The  major  questions  in  determining  how  to  best  present  the  data  were  the  amount  of  data 
available  and  how  to  make  the  information  presentable  and  understandable  without  losing  the 
nuances  and  meaning  contained  in  the  data.  Large  matrices  of  the  raw  number  of  responses  in 
different  categories  were  unlikely  to  adequately  convey  the  meaning  of  the  results.  Throughout 
the  analysis  chapters,  when  there  was  any  question  as  to  whether  to  include  a  chart  or  graph  that 
might  aid  the  reader  in  understanding  the  analysis,  the  chart  was  included.  In  the  summary 
chapters,  much  of  this  detailed  information  is  omitted  and  the  reader  is  referred  to  the  earlier 
chapters. 

Figure  7-6  shows  the  main  format  used  throughout  the  thesis  to  present  the  results.  It 
combines  several  types  of  information.  First,  the  various  factors  are  plotted  by  their  mean 
values.  About  each  mean  is  plotted  the  standard  error  of  the  mean  based  on  the  variance  in  the 
data.  Superimposed  on  this  are  the  statistical  groups,  as  determined  by  the  Wilcoxon  signed-rank 
test.  This  chart  represents  many  compromises,  but  was  determined  to  be  the  most  acceptable, 
meaningful,  and  understandable  method  for  presenting  the  large  amounts  of  data  collected. 
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Figure  7-5:  Example  of  a  Statistical-Grouping  Chart  Overlaid  on  Average  Responses  and 
Standard  Errors 


Other  types  of  charts  used  to  present  the  data  include  the  distribution  of  raw  responses, 
and  displays  of  differences  between  two  factors.  Charts  showing  the  latter  are  used  sparingly,  as 
they  show  the  difference  between  only  two  factors  at  a  time.  Displaying  pairs  of  charts  for  all 
interested  pairs  of  responses  would  be  prohibitive. 

These  types  of  charts  are  intended  to  convey  the  thesis  message  clearly  to  a  wide  range  of 
audiences.  Each  chart  loses  some  information  contained  in  the  data,  and  thus  caution  must  be 
taken  in  interpreting  the  charts. 
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Chapter  Summary 

Throughout  the  surveys  various  types  of  information  was  collected.  To  analyze  and 
present  the  results  of  the  surveys  accepted  statistical  methods  have  been  used.  To  analyze  the 
interval  scale  data  the  Wilcoxon  Signed  Rank  test  was  selected  because,  as  a  non-parametric 
statistical  test,  it  did  not  rely  on  assumptions  of  normal  distributions  and  was  considered  more 
powerful  in  determining  statistical  differences  than  the  sign  test.  This  method  is  used  to  compare 
responses  within  each  individual  survey  and  then  to  look  at  the  distribution  of  the  differences 
across  all  surveys.  From  this  statistical  groups  for  the  different  variables  were  established  and  a 
graphical  method  of  presentation  was  introduced. 


Part  3 


The  Development  of  a  Project’s  Initial  Schedule 
and  Its  Impacts :  Survey  Results  and  Analysis 


Part  3:  Overview 


Part  3  describes  the  processes  used  to  develop,  contract,  and  execute  project  schedules 
based  on  the  317  surveys  received.  Chapter  8  describes  the  development  of  a  project’s  initial 
schedule  by  analyzing  the  survey  results.  Chapter  9  documents  the  effects  of  the  initial  project’s 
schedule  on  the  contracting  phase.  It  shows  that  the  initial  schedule  has  the  dominant  impact  on 
contractors’  proposed  schedules  and  the  resulting  contracted  schedules.  Chapter  10  identifies 
schedule-related  incentives  for  users,  the  Pentagon,  the  Program  Offices,  and  defense  contractors 
during  the  development  phase.  Chapter  1 1  highlights  the  effects  of  a  project’s  initial  schedule 
and  the  subsequent  contracted  schedule  on  the  actual  development  schedule.  It  compares  the 
project’s  achieved  schedule  with  the  planned  schedule.  Part  3  provides  the  foundation  from 
which  Part  4  draws  conclusions  and  makes  specific  policy  recommendations. 
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Chapter  8 


Development  of  a  Project’s  Initial  Schedule 


The  development  of  a  project’s  initial  schedule  occurs  early  in  the  acquisition  planning 
phase  and  prior  to  project  approval.  Once  established,  the  project’s  initial  schedule  forms  a  basis 
for  the  rest  of  the  acquisition  planning.  As  will  be  shown  in  following  chapters,  the  initial 
project  schedule  plays  a  central  role  in  the  time  used  to  develop  new  products  and  systems. 


This  chapter  presents  the  results  of  the  three  surveys  described  in  Chapter  6  as  they  pertain  to 
development  of  a  project’s  initial  schedule.  The  areas  evaluated  include  the  user’s  desired 
schedule,  the  priority  assigned  to  a  project’s  schedule,  the  factors  that  determine  a  project’s 


starting  date,  the  organizations  involved  in  developing  a  project’s  schedule,  and  the  factors  that 
influence  the  length  of  a  project’s  schedule.  Also  evaluated  are  information  and  tools  used  to 
develop  a  project’s  schedule. 


Figure  8-1:  A  Project’s  Initial  Schedule,  as  Developed  during  the  Project’s  Planning  Phase 
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A.  When  the  User  Desires  a  Project 

When  establishing  a  project’s  initial  schedule,  a  primary  consideration  is  when  the  user 
desires  the  system.  The  Pentagon  survey  asked  program  element  monitors  and  Pentagon  action 
officers  the  date  by  which  users  wanted  the  system  and  whether  it  was  meant  to  meet  a  current  or 
a  future  operational  deficiency.  The  respondents  indicated  that  80  percent  of  users  desired 
systems  "as  soon  as  possible,”  with  20  percent  listing  a  specific  date.  Respondents  also  referred 
to  items  such  as  the  need  to  integrate  the  system  with  others  in  development,  launch  schedule 
dates,  and  projected  emerging  threats  as  determining  the  desired  date. 

User’s  Desired  Date 


“ASAP” 


Specified 
Future  Date 


Program  Element  Monitor  Survey 

Figure  8-2:  User’s  Desired  Date  of  First  Production  Item,  as  Reported  by  Program 
Element  Monitors  (Number  of  Projects  =  62) 


Seventy  percent  of  respondents  indicated  that  the  system  was  intended  to  meet  a  current 
operational  deficiency,  while  30  percent  indicated  that  the  system  was  intended  to  meet  a  future 
or  projected  need. 
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Operational  Need  Date 


Current 

Operational 

Deficiency 


Future  or 
Projected 
Deficiency 


Figure  8-3:  Operational  Need  Date  for  a  System  in  Development,  as  Reported  by  Program 
Element  Monitors  (Number  of  Projects  =  62) 


The  responses  indicated  that  for  most  development  efforts,  the  longer  the  time  required  to 
develop  the  products,  the  longer  a  need  would  go  unfilled.  A  project  schedule  may  also  depend 
on  the  priority  users  give  to  a  project.  Both  the  Program  Office  and  the  Pentagon  surveys  asked 
respondents  to  indicate  the  priority  assigned  by  the  user  commands,  on  a  scale  of  1  to  7.  Thirty- 
one  percent — or  65  of  209  respondents-indicated  that  their  project  was  the  highest  priority. 

Over  60  percent  selected  one  of  the  two  highest  responses.  Only  27  percent  indicated  that  their 
project  was  average  or  lower  among  the  user’s  priorities. 

Based  on  the  expressed  need,  users’  desired  date  of  “as  soon  as  possible,”  and  the 
reported  high  priority  users  placed  on  projects,  a  short  development  schedule  would  seem  to  be  a 
significant  priority.  However,  that  does  not  appear  to  be  the  case. 
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User  Project  Priority 

Figure  8-4:  Project  Priority  for  the  User  of  a  System  (Program  Office  and  Pentagon 
Surveys;  Number  of  Projects  =  209) 


B.  Priority  of  Schedule  in  Development  Projects 

To  evaluate  the  priority  assigned  to  short  schedules  in  the  program  planning  phase,  the 
Program  Office  and  the  Pentagon  surveys  asked  respondents  to  rank  four  project  priorities:  low 
acquisition  cost,  low  operational  and  support  costs,  short  schedule  to  operational  capability,  and 
superior  technical  performance.  Superior  technical  performance  was  most  often  rated  as  a 
project’s  highest  priority.  Low  acquisition  cost  was  most  often  rated  as  a  project’s  second 
priority.  Low  operational  cost  was  most  often  rated  as  a  project’s  third  priority.  Shortened 
schedule  was  most  often  rated  as  a  project’s  fourth-or  lowest-priority. 

Figure  8-6  shows  the  number  of  respondents  rating  each  objective  first,  second,  third,  or 
fourth.  Among  208  projects,  superior  performance  was  rated  as  the  first  objective  in  93  projects, 
low  acquisition  cost  was  rated  second  objective  in  7 1  projects,  low  operating  cost  were  rated 
third  in  75  projects,  and  short  schedule  was  rated  the  lowest  objective  in  78  projects.  No 
significant  differences  were  noted  between  the  Program  Office  and  Pentagon  surveys. 
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Figure  8-5:  Ranking  of  Project  Objectives  by  Program  Managers  and  Program  Element 
Monitors  (Program  Office  and  Pentagon  Surveys;  Number  of  Projects  =  208) 


Comparing  the  responses  on  a  survey-by-survey  basis  using  the  paired-samples  sign  test 
indicates  very  significant  differences  among  the  different  objectives.  Superior  performance  rated 
statistically  higher  than  all  other  objectives,  with  a  confidence  level  of  over  99  percent.  Superior 
performance  rated  higher  than  shortened  schedule,  with  a  confidence  level  of  99.99  percent. 

Low  acquisition  cost  rated  higher  than  shortened  schedules,  with  a  confidence  level  of  97.5 
percent.  Low  operating  cost  rated  higher  than  shortened  schedule,  but  with  a  confidence  level  of 
only  91  percent.  Table  8-1  shows  the  statistical  significance  of  the  differences  among  the  various 
project  objectives. 


1 90  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Program  Objective 
Significance  Table 


Perform  AcqCost  Ops  Cost  Schedule 


Table  8-2:  Significance  of  the  Differences  among  Project  Objectives  Using  the  Non- 
Parametric  Paired-Samples  Sign  Test  (Program  Office  and  Pentagon  Surveys;  Number  of 
Projects  =  209) 

Comparing  short  project  schedules  with  other  objectives  does  not  completely  reveal  the 
importance  of  development  time,  but  does  indicate  its  relative  position.  To  determine  the 
importance  of  short  schedules,  the  Pentagon  and  Program  Office  surveys  asked  to  what  extent  a 
short  acquisition  cycle  was  an  important  part  of  a  project’s  overall  objectives.  Among  the  206 
respondents,  less  than  15  percent  indicated  that  a  quick  acquisition  cycle  was  “very  important.” 
One-third  of  the  respondents  indicated  that  short  acquisition  time  was  a  significant  priority  by 
selecting  one  of  the  two-highest  categories.  Over  half  the  respondents  indicated  that  it  was  a 
“somewhat  important”  or  “not  important”  objective. 

The  low  priority  assigned  to  schedule  is  consistent  with  the  finding  that  only  27  percent 
of  projects  are  considered  “fast  track”--  having  shorter-than-normal  planned  development  times. 
Seventy-three  percent  are  not  considered  “fast  track,”  and  do  not  attempt  to  shorten  the  normal 
development  schedule. 
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Figure  8-6:  Short  Acquisition  Cycle  as  a  Program  Objective,  Reported  by  Project 
Managers  and  Program  Element  Monitors  (Program  Office  and  Contractor  Surveys; 
Number  of  Projects  =  206) 

Fast  Track  Vs  Normal  Projects 


“Fast  Track” 


Figure  8-7:  Projects  Reportedly  on  a  Fast  Track-Having  an  Officially  Accelerated 
Acquisition  Cycle  (Program  Office,  Pentagon,  and  Contractor  Surveys;  Number  of 
Projects  =  317) 
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C.  Determining  a  Project’s  Starting  Date 

The  Program  Office  and  Pentagon  surveys  asked  a  series  of  questions  to  determine  which 
organizations  are  responsible  for  initiating  projects,  and  what  factors  are  considered  in 
establishing  a  project’s  starting  date. 

The  Program  Office  survey  asked  program  managers  to  identify  the  organizations  that 
initially  promoted  the  project  concept  or  provided  funding.  They  reported  that  the  services’  user 
commands  (48  percent)  or  service  headquarters  (21  percent)  initiated  the  majority  of  current 
projects.  A  smaller  percentage  of  projects  were  started  by  development  and  logistics  centers  (9 
percent),  other  services  (4  percent),  contractors  (4  percent),  elected  or  appointed  government 
officials  (3  percent),  the  Defense  Advanced  Projects  Agency,  (>1  percent),  or  other  entities  (10 
percent).  The  individual  services,  through  the  users,  service  headquarters,  and  development  and 
logistics  centers,  initiated  at  least  78  percent  of  the  projects  surveyed. 

To  understand  the  process  used  to  select  the  projects,  the  Pentagon  Survey  asked  if  a 
project  was  initiated  through  the  formal  modernization  planning  process  or  through  specific 
leadership  direction.  As  described  in  Chapter  3,  the  modernization  planning  process  is  intended 
to  identify  and  prioritize  new  systems  and  modifications  needed  to  meet  operational  requirements 
in  each  mission  area.  Survey  results  indicated  that  58  percent  of  the  projects  were  initiated 
through  the  modernization  planning  process,  while  42  percent  of  all  projects  were  initiated 
outside  that  process  through  senior  leadership  direction. 
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Figure  8-8:  Number  of  Projects  Initiated  by  Various  Organizations  (Program  Office 
Survey;  Number  of  Projects  =  129) 


Project  Initiation  Source 


Leadership 

Directed 

Project 


Program  Element  Monitor  Survey 

Figure  8-9:  Percentage  of  Projects  Started  through  the  Modernization  Planning  Process 
and  Senior  Leadership  Direction  (Pentagon  Survey;  Number  of  Projects  =  60) 
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The  Pentagon  Survey  asked  respondents  to  rate  factors  thought  to  influence  starting  dates 
for  their  projects  on  a  scale  of  1  to  7,  from  “no  impact  to  defining  impact.  Although  caution 
must  be  taken  when  using  average  data,  they  do  shed  light  on  the  influence  of  various  factors. 
Besides  the  average  responses,  the  factors  were  also  grouped  statistically  using  the  Wilcoxon 
ranked  sign  test,  a  paired-sample  non-parametric  test. 

The  remits  show  that  the  primary  influences  in  setting  a  project’s  starting  date  are  the 
user’s  desires  and  the  desires  of  the  service  leadership.  These  factors  were  rated  significantly 
higher  than  any  other  factor.  They  were  followed  by  funding-related  aspects,  including  the 
expected  availability  of  development  funding,  the  expected  availability  of  production  funding, 
and  the  results  of  the  service  planning  process,  which  allocates  resources. 
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Figure  8-10:  Impact  of  Various  Factors  in  Determining  a  Project’s  Starting  Date 
(Pentagon  Survey  Number  of  Projects  =  52;  Error  Bars  =  +/-1  SE96) 


96  Note:  The  error  bars  shown  are  determined  using  plus  or  minus  one  standard  error  of  the  mean  and  the  statistical 
groups  are  determined  using  the  Wilcoxon  non-parametric  statistical  test  as  described  in  Chapter  7. 
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The  remaining  factors  form  a  third  statistical  group  that  includes  circumstances  typically 
associated  with  actual  requirements:  technology  development,  engineering  development,  and 
development  of  the  manufacturing  process.  This  group  also  includes  testing  requirements, 
support  requirements,  and  dependence  on  other  programs.  The  Program  Element  Monitors  and 
Pentagon  action  officers  report  that  these  factors  have  a  significantly  smaller  impact*  The  user’s 
desires  were  reported  to  have  a  larger  impact  6  times  more  often  than  technology  development 
(30  projects  to  5  projects).  (Technology  development  was  the  highest-rated  development-related 
factor.)  Fifteen  projects  reported  the  two  factors  as  tied.  Table  8-2  shows  the  number  of  projects 
for  which  the  user’s  desires  were  rated  as  having  a  larger  impact  than  other  factors. 


The  reported  relative  impact  of  the  user’s 
desires  vs.  other  factors  on  the  project 
starting  date 

Other  Factor 

Number  of 
Projects 
Rating 
User’s 
Desires 
Higher 

Number  of 
Projects 
Rating 
Other 
Factor 
Higher 

Number  of 
Projects 
Rating 
Them 
Equal 

Service  Leadership  Desires 

20 

10 

21 

Expected  Development  Funding 

24 

10 

16 

Expected  Production  Funding 

25 

5 

18 

Service  Planning 

30 

8 

1  11 

Technology  Development 

30 

5 

15 

Dependence  on  Another  Program 

29 

6 

15 

Engineering  Requirements 

31 

4 

15 

Testing  Requirements 

35 

1 

14 

Manufacturing  Requirements 

31 

6 

16 

Support  Requirements 

36 

2 

12 

Table  8-3:  Reported  Influence  of  User’s  Desires  vs.  Other  Factors  in  Determining  a 
Project’s  Starting  Date  (Pentagon  Survey;  Number  of  Projects  =  51) 


Once  a  project  is  started,  planning  activities  used  to  manage  the  development  begin.  One 
aspect  of  planning  is  development  of  the  project’s  initial  schedule.  The  parties  that  develop  the 
project’s  initial  schedule,  and  the  factors  influence  its  length,  will  be  addressed  next. 
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D.  The  Parties  Involved  in  Developing  a  Project’s  Initial  Schedule 

The  Program  Office  and  Pentagon  surveys  asked  respondents  to  indicate  the  involvement 
and  influence  of  various  organizations  in  developing  their  project’s  initial  schedule.  This  was 
done  using  a  7-point  scale  ranging  from  “not  involved”  to  “extensively  involved.  The  responses 
show  that  in  a  majority  of  projects.  Program  Offices  are  significantly  more  involved  in 
developing  the  initial  schedule  than  any  other  group.  The  users  are  the  second  most  involved, 
while  potential  contractors  were  reported  third.  Program  Offices  were  reported  to  be  more 
involved  in  developing  the  initial  schedule  in  5  times  as  many  projects  as  the  next-highest  group, 
the  users.  (The  figures  were  70  versus  14  projects,  with  52  reporting  the  two  groups  as  equally 
involved.)  Figure  8-12  and  Table  8-3  compare  Program  Office  involvement  and  the  involvement 
of  other  organizations. 
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Users 

Potential  Contractors 
Service  Acquisition  Community 
Service  Budget  Community 
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FFRDC 

DoD  Budget  Community 
DoD  Planning  Community 


Mean  +/- 1  SE  SPOs  only 


Figure  8-11:  Organizational  Involvement  in  Setting  the  Initial  Project  Schedule  (Program 
Office  Survey;  Number  of  Projects  =  137) 
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The  relative  involvement  of  the 

Number  of 

Number  of 

Program  Offices  in  establishing  the 

Projects  Rating 

Projects  Rating 

of 

initial  schedule  vs.  the  involvement 

Program  Office 

the  Involvement 

Projects 

of  other  organizations 

Involvement 

of  Other 

Rating 

Higher 

Organizations 

Them 

Other  organizations 

Higher 

Equal 

Users 

70 

14 

52 

Potential  Contractors 

76 

14 

45 

Service  Acquisition  Community 

105 

9 

20 

Service  Budget  Community 

109 

7 

18 

Potential  Sub-Contractors 

105 

9 

15 

PEO 

110 

6 

16 

DoD  Acquisition  Community 

113 

8  n 

12 

FFRDC 

103 

8 

20 

DoD  Budget  Community 

114 

8 

10 

DoD  Planning  Community 

118 

6 

7 

Center  Headquarters 

111 

3 

17 

Table  8-4:  Involvement  of  the  Program  Office  vs.  Other  Organizations  in  Establishing  a 
Project’s  Initial  Schedule  (Program  Office  Survey;  N=137) 


Involvement  in  the  process  to  develop  a  project’s  initial  schedule  does  not  necessarily 
imply  influence.  Pentagon  survey  respondents  were  asked  to  rank  the  influence  of  various 
groups  on  their  project’s  initial  schedule  on  a  7-point  scale,  from  “not  involved”  to  “sole 
influence.”  They  reported  that  the  Program  Office  and  the  users  had  the  most  influence,  with 
those  groups  rating  significantly  higher  than  all  other  organizations.  The  next  group  of 
organizations  reportedly  influencing  a  project’s  initial  schedule  are  the  service  acquisition 
community  and  the  service  leadership.  (This  is  more  influence  than  would  have  been  expected 
from  the  results  of  the  Program  Office  survey.) .  The  influence  of  other  organizations,  including 
potential  contractors,  the  services,  and  DoD  planning  and  budgeting  communities,  was  rated 
significantly  lower  than  that  of  the  first  four  groups.  Figure  8-14  shows  the  reported  influence  of 
the  various  organizations  across  all  projects  included  in  the  Pentagon  survey.  Again,  the 
Wilcoxon  ranked-sign  method  was  used  to  group  factors  statistically.  Table  8-5  compares  the 
reported  influence  of  the  Program  Office  with  that  of  other  organizations. 
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Program  Offices 
Users 

Service  Acquisition  Community 
Service  Leadership 
Potential  Contractors 
DoD  Leadership 
DoD  Acquisition  Community 
Service  Budget  Community 
PEO 

DoD  Budget  Community 
Service  Planning  Community 
DoD  Planning  Community 
FFRDC 

Potential  Sub-Contractors 
Service  Cost  Analysis  Group 


Mean  +/- 1  SE  Pentagon  Survey 


Figure  8-12:  Organizational  Influence  in  Setting  the  Initial  Project  Schedule,  as  Reported 
by  Pentagon  Respondents  (Number  of  Projects  =  52) 


The  relative  influence  of  Program 
Offices  vs.  other  organizations  in 
establishing  a  project’s  initial 
schedule 

Other  organizations 

Number  of 
Projects 

Rating 

Program 

Office 

Influence 

Higher 

Number  of 
Projects  Rating 
Other 

Organizations’ 

Influence 

Higher 

Number 

of 

Projects 

Rating 

Them 

Equal 

Users 

17 

14 

21 

Service  Acquisition  Community 

26 

5 

21 

Service  Leadership  Community 

29 

6 

17 

Potential  Contractors 

34 

4 

12 

DoD  Leadership 

35 

7 

10 

DoD  Acquisition  Community 

35 

3 

13 

Table  8-5:  Reported  Influence  of  Program  Offices  versus  Other  Organizations  in 
Determining  the  Initial  Project  Schedule  (Pentagon  Survey;  Number  of  Projects  =  52) 
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The  Program  Offices  reported  significant  involvement  by  contractors  in  the  initial  project 
schedule,  but  Pentagon  respondents  indicated  that  defense  contractors  had  significantly  less 
influence  than  the  service  acquisition  community  and  the  service  leadership.  (In  the  Pentagon 
survey,  48  percent  of  respondents  indicated  that  the  service  acquisition  community  had  more 
influence  than  defense  contractors,  while  24  percent  reported  that  the  defense  contractors  had 
more  influence  than  the  service  acquisition  community.  Twenty-eight  percent  reported  the 
influence  of  the  two  organizations  as  the  same.)  It  is  unclear  whether  these  results  reflect 
differences  in  organizational  perspectives. 

As  would  be  expected,  the  Office  of  the  Secretary  of  Defense  was  more  involved  in  the 
largest  development  programs  than  in  smaller  programs.  However,  even  for  large  programs,  the 
involvement  and  influence  of  OSD  organizations  was  still  much  below  those  of  service  users  and 
Program  Offices. 

The  analysis  indicates  that  the  primary  organizations  involved  in  developing  project 
schedules  are  the  Program  Offices  and  the  users.  Potential  contractors  are  often  involved  but 
have  less  influence.  The  planning,  budget,  and  cost  analysis  groups,  federally  funded  research 
and  development  centers,  and  potential  subcontractors  do  not  play  a  significant  role,  nor  does 
DoD  for  most  projects. 


E.  The  Factors  Determining  the  Length  of  a  Project’s  Initial  Schedule 

To  determine  the  influences  on  length  of  a  project’s  initial  schedule,  the  Program  Office 
and  the  Pentagon  Surveys  asked  respondents  to  rate  a  number  of  factors  identified  during 
preliminary  interviews  and  literature  research.  The  respondents  rated  these  factors  on  a  scale  of  1 
to  7,  ranging  from  “no  impact”  to  “defining  impact.”  The  average  responses  provide  an 
understanding  of  the  relative  importance  of  different  factors  across  all  the  projects.  The 
Wilcoxon  ranked-sign  test  was  also  used  to  identify  groups  of  factors  that  can  be  separated  from 
the  others  on  a  statistical  basis. 

The  factors  reportedly  having  the  largest  impact  are  the  user’s  desires  and  the  availability 
of  development  funding.  These  factors  were  rated  statistically  higher  than  all  other  factors.  They 
were  followed  by  the  availability  of  expected  production  funding.  The  factors  typically 
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associated  with  development-related  aspects— engineering  requirements,  technological 
development,  and  development  of  the  manufacturing  process— rated  significantly  lower,  ranking 
fourth,  seventh,  and  tenth  among  11  factors,  respectively. 
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Figure  8-13:  Factors  Affecting  the  Length  of  a  Project’s  Initial  Schedule  (Pentagon  and 
Program  Office  Surveys;  Number  of  Projects  =  178) 


Both  the  Program  Office  and  the  Pentagon  surveys  rated  users’  desires  and  the 
availability  of  development  and  production  funding  as  having  the  largest  impact.  Notable 
differences  between  the  two  surveys  did  appear,  such  as  in  the  ranking  of  testing  requirements 
and  service  planning.  But  both  surveys  reported  technology  development  and  the  manufacturing 
process  as  having  a  low  reported  impact  on  the  length  of  the  development  schedule. 
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No  Impact  Some  Impact  Defining  Impact 


User’s  Desires 
Expected  Development  Funding 
Expected  Production  Funding 
Testing  Requirements 
Engineering  Requirements 
Service  Leadership  Desires 
Depend  on  Another  Program 
Technology  Development 
Service  Planning 
Support  Requirements 
Manufacturing  Process  Dev 
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Mean  +/- 1  SE  SPOs  Only 

Figure  8-14:  Average  Rating  for  Factors  Used  to  Determine  the  Length  of  Initial  Project 
Schedule  (Program  Office  Survey;  Number  of  Projects  =  126) 


No  Impact  Some  Impact  Defining  Impact 

User’s  Desires 
Expected  Development  Funding 
Expected  Production  Funding 
Service  Leadership  Desires 
Engineering  Requirements 
Service  Planning 
Technology  Development 
Manufacturing  Process  Dev 
Testing  Requirements 
Depend  on  Another  Program 
Support  Requirements 

1  2  3  4  5  6  7 

Mean  +/- 1  SE  Pentagon  Only 

Figure  8-15:  Average  Rating  for  Factors  Used  to  Determine  the  Length  of  Initial  Project 
Schedule  (Pentagon  Survey;  Number  of  Projects  =  52) 
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Averaging  the  data  may  mask  important  aspects  of  the  information.  Analyzing  the 
responses  on  a  factor-by-factor  and  project-by-project  basis  makes  clear  that  user’s  desires, 
expected  development  funding,  and  expected  production  funding  have  a  larger  impact  on  most 
projects  than  -  other  factors.  The  tables  on  the  next  page  show  the  project-by-project  comparison 
of  responses  regarding  the  user’s  desires,  expected  development  funding,  and  expected 
production  funding  versus  other  factors. 

The  user’s  desired  date  and  expected  development  funding  are  rated  more  important  on 
approximately  the  same  number  of  projects.  For  example,  69  projects  rated  the  user’s  desires  as 
more  important  than  expected  development  funding,  while  64  projects  rated  expected 
development  funding  higher  than  the  user’s  desires.  Forty-four  projects  rated  them  equal.  These 
responses  cannot  be  statistically  separated.  In  contrast,  the  user’s  desires  were  rated  as  having 
more  influence  on  twice  as  many  projects  (88  to  42)  as  engineering  requirements. 

When  viewed  across  all  projects,  expected  development  funding  has  a  larger  impact  on 
the  length  of  the  initial  project  schedule  than  any  development-related  aspects,  including 
technology  development,  engineering  requirements,  and  development  of  the  manufacturing 
process.  For  example,  expected  development  funding  had  a  larger  impact  than  engineering 
requirements  in  88  projects,  and  a  smaller  impact  than  engineering  requirements  in  57  projects. 
Expected  development  funding  had  a  larger  than  technology  development  in  100  projects,  and  a 
smaller  impact  in  42  projects.  Expected  development  funding  had  a  larger  impact  than 
manufacturing  process  development  in  118  projects,  while  ranking  lower  in  26  projects.  These 
differences  are  significant  based  on  the  non-parametric  paired-sample  sign  test  at  the  0.01 
percent,  1  percent  and  0.01  percent  levels,  respectively.  Clearly  the  expected  development 
funding  plays  a  major  role  in  the  length  of  a  project’s  initial  schedule. 
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Factors  Influencing  the  Length  Project  Schedule 
User’s  Desires  vs.  Other  Factors 


The  reported  relative  impact  of  the  user’s 
desired  schedule  vs.  other  factors  on  the 
length  of  a  project’s  initial  schedule 

Other  Factors 

Number  of 

Projects  Rating 
User’s  Desires 
Higher 

Number  of 

Projects  Rating 

Other  Factors 

Higher 

Number  of 
Projects 

Rating 

Them  Equal 

Expected  Development  Funding 

69 

64 

52 

Expected  Production  Funding 

78 

53 

41 

Engineering  Requirements 

88 

42 

47 

Testing  Requirements 

98 

42 

38  & 

Service  Leadership  Desires 

87 

20 

69 

Technology  Development 

105 

38 

34 

Service  Planning 

96 

24 

53 

Dependence  on  Another  Program 

109 

39 

25 

Manufacturing  Process  Development 

118 

26 

31 

Support  Requirements 

129 

20 

26 

Expected  Development  Funding  vs.  Other  Factors 


The  reported  relative  impact  of  expected 
development  funding  vs.  other  factors  on  the 
length  of  the  initial  schedule 

Other  Factors 

Number  of 

Projects  Rating 
Expected  Dev 
Funding  Higher 

Number  of 

Projects  Rating 

Other  Factors 

Higher 

Number  of 
Projects 

Rating 

Them  Equal 

User’s  Desires 

64 

69 

52 

Expected  Production  Funding 

41 

24 

108 

Engineering  Requirements 

88 

57 

33 

Testing  Requirements 

101 

51 

27 

Service  Leadership  Desires 

91 

45 

41 

Technology  Development 

100 

42 

35 

Service  Planning 

104 

38 

32 

Dependence  on  Another  Program 

96 

44 

33 

Manufacturing  Process  Development 

118 

26 

32 

Support  Requirements 

121 

29 

25 

Expected  Production  Funding  vs.  Other  Factors 


The  reported  relative  impact  of  expected 
production  funding  vs.  other  factors  on  the 
length  of  a  project’s  initial  schedule 

Other  Factors 

Number  of  * 
Projects  Rating 
Expected  Prod 
Funding  Higher 

Number  of 

Projects  Rating 

Other  Factors 

Higher 

Number  of 
Projects 

Rating 

Them  Equal 

User’s  Desires 

53 

78 

41 

Expected  Development  Funding 

24 

41 

108 

Engineering  Requirements 

78 

60 

35 

Testing  Requirements 

83 

55 

36 

Service  Leadership  Desires 

79 

50 

43 

Technology  Development 

88 

47 

37 

Service  Planning 

85 

38 

46 

Dependence  on  Another  Program 

85 

47 

36 

Manufacturing  Process  Development 

99 

20 

47 

Support  Requirements 

103 

27 

40 

Tables  8-6, 8-7,  and  8-8:  Number  of  Projects  Reporting  the  Impact  of  the  User’s  Desires, 
Expected  Development  Funding,  and  Expected  Production  Funding  as  Larger,  Smaller,  or 
Equal  to  Other  Factors  in  Determining  Length  of  a  Project’s  Initial  Schedule 
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These  results  appear  to  contradict  earlier  results  indicating  that  users  wanted  80  percent  of 
projects  as  soon  as  possible.  This  contradiction  was  discussed  during  presentation  of  the  data  at 
the  Aeronautical  Systems  Center,  the  Electronic  Systems  Center,  the  Pentagon,  and  MIT.  The 
strong  consensus  of  participants  at  each  location  was  that  users’  desires  were  determined 
primarily  by-expected  development  and  production  funding-not  their  actual  operational  needs. 
This  is  also  consistent  with  results  from  other  survey  questions. 

In  a  related  question,  the  Pentagon  survey  asked  respondents  to  choose  between  two 
factors  limiting  their  project’s  schedule:  funding,  and  technology  and  engineering.  Seventy- 
seven  percent  indicated  that  funding  was  the  limiting  factor,  supporting  the  earlier  conclusions. 


Funding  Limited  Vs 
Technology  and  Engineering 


Funding 

Limited 


Figure  8-16:  Percent  of  Respondents  Reporting  the  Limiting  Factor  for  Their  Project’s 
Schedule  as  Funding  or  Technology  and  Engineering  (Pentagon  Survey;  Number  of 
Projects  =  61) 

The  finding  that  expected  funding  availability  has  a  larger  impact  than  development- 
related  factors  is  consistent  across  all  types,  sizes,  and  technological  levels  of  projects.  This 
consistency  indicates  that  the  finding  is  systemic  across  the  entire  range  of  projects  and  the  entire 
development  process,  and  cannot  be  dismissed  as  resulting  from  specific  circumstances.  The 
following  graphs  show  the  average  reported  impact  of  each  factor  on  different  subgroups  of 
projects. 


8.  Development  of  a  Project’s  Initial  Schedule 


205 


The  Impact  of  Expected  Development  Funding  vs.  Engineering  Requirements  on  Length  of 

a  Project’s  Initial  Schedule 


#  Projects  Rating 
Expected 

Development 

Funding  More 
Important 

#  Projects  Rating 

Engineering 

Requirements 

More  Important 

#  Rating 
Them 

Equally 

Important 

Total  Number 

All  Projects 

88 

57 

33 

178 

By  System  Type 

#  Projects  Rating 
Expected 

Development 

Funding  More 
Important 

#  Projects  Rating 

Engineering 

Requirements 

More  Important 

#Rating 

Them 

Equally 

Important 

Total  Number 

Aircraft 

35 

24 

14 

73 

Spacecraft 

5 

3 

0 

8 

Electronics 

10 

5 

5 

20 

Missiles  and  Munitions 

15 

6 

3 

24 

Software 

13 

10 

3 

26 

Other 

9 

8 

6 

23 

By  Project  Size 

#  Projects  Rating 
Expected 

Development 

Funding  More 
Important 

#  Projects  Rating 

Engineering 

Requirements 

More  Important 

#  Rating 
Them 
Equally 
Important 

Total  Number 

ACAT I  Programs 

32 

18 

9 

59 

ACAT II  Programs 

14 

10 

6 

30 

ACAT  III  Programs 

42 

28 

17 

86 

By  Technological  Advance 

#  Projects  Rating 
Expected 

Development 

Funding  More 
Important 

#  Projects  Rating 

Engineering 

Requirements 

More  Important 

#Rating 

Them 

Equally 

Important 

Total  Number 

Revolutionary  Product 

10 

8 

1 

19 

New  Generation  Product 

30 

22 

10 

62 

Incremental  Improvement 

46 

27 

22 

95 

By  Organization 

#  Projects  Rating 
Expected 

Development 

Funding  More 
Important 

#  Projects  Rating 

Engineering 

Requirements 

More  Important 

#Rating 

Them 

Equally 

Important 

Total  Number 

Pentagon 

25 

16 

ii 

52 

Program  Office 

63 

41 

22 

126 

Table  8-9:  Number  of  Projects  Reporting  the  Impact  of  Expected  Development  Funding  as 
Larger,  Smaller,  or  Equal  to  Engineering  Requirements  in  Determining  Length  of  Initial 
Schedule  (Pentagon  and  Program  Office  Surveys:  Number  of  Projects  =  178) 
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Impact  of  Expected  Development  Funding  vs.  Technology  Development  on  Length  of 

Initial  Project  Schedule 


Expected 
Development 
Funding  Rated 

More  Important 

Technology 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

All  Projects 

100 

42 

35 

177 

By  System  Type 

Expected 
Development 
Funding  Rated 

More  Important 

Technology 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Aircraft 

43 

17 

13 

73 

Spacecraft 

5 

3 

0 

8 

Electronics 

10 

5 

5 

20 

Missiles  and  Munitions 

14 

4 

6 

24 

Software 

15 

6 

5 

26 

Other 

11 

7 

5 

23 

By  Project  Size 

Expected 
Development 
Funding  Rated 

More  Important 

Technology 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

ACAT I  Programs 

36 

12 

ii 

59 

ACAT II  Programs 

16 

6 

8 

30 

ACAT  III  Programs 

48 

23 

15 

86 

By  Technological  Advance 

Expected 
Development 
Funding  Rated 

More  Important 

Technology 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Revolutionary  Product 

9 

8 

2 

19 

New  Generation  Product 

35 

17 

10 

62 

Incremental  Improvement 

55 

17 

23 

95 

By  Organization 

Expected 
Development 
Funding  Rated 

More  Important 

Technology 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Pentagon 

26 

16 

9 

52 

Program  Office 

74 

26 

26 

!  126 

Table  8-10:  Number  of  Projects  Reporting  the  Impact  of  Expected  Development  Funding 
as  Larger,  Smaller,  or  Equal  to  Technology  Development  in  Determining  Length  of  Initial 
Schedule  (Pentagon  and  Program  Office  Surveys;  Number  of  Projects  =  178) 


8.  Development  of  a  Project’s  Initial  Schedule 


207 


Impact  of  Expected  Development  Funding  vs.  Manufacturing  Process  on  Length  of  Initial 

Project  Schedule 


Expected 
Development 
Funding  Rated 

More  Important 

Manufacturing 

Process 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

All  Projects 

118 

26 

32 

176 

By  System  Type 

Expected 
Development 
Funding  Rated 

More  Important 

Manufacturing 

Process 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Aircraft 

43 

13 

16 

72 

Spacecraft 

5 

1 

2 

8 

Electronics 

13 

2 

6 

21 

Missiles  and  Munitions 

18 

4 

2 

24 

Software 

20 

3 

2 

25 

Other 

18 

2 

3 

23 

By  Project  Size 

Expected 
Development 
Funding  Rated 

More  Important 

Manufacturing 

Process 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

ACAT I  Programs 

38 

9 

12 

59 

ACAT II  Programs 

20 

3 

6 

29 

ACAT  III  Programs 

60 

13 

13 

86 

By  Technological  Advance 

Expected 
Development 
Funding  Rated 

More  Important 

Manufacturing 

Process 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Revolutionary  Product 

12 

3 

4 

19 

New  Generation  Product 

41 

11 

10 

62 

Incremental  Improvement 

64 

12 

17 

93 

By  Organization 

Expected 
Development 
Funding  Rated 

More  Important 

Manufacturing 

Process 

Development 

Rated  More 
Important 

Rated 

Equally 

Important 

Total  Number 

Pentagon 

29 

12 

ii 

52 

Program  Office 

89 

12 

21 

124 

Table  8-11:  Number  of  Projects  Reporting  the  Impact  of  Expected  Development  Funding 
as  Larger,  Smaller,  or  Equal  to  Manufacturing  Process  in  Determining  Length  of  Initial 
Schedule  (Pentagon  and  Program  Office  Surveys;  Number  of  Projects  =  178) 
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Factors  Affecting  the  Length  of  the  Initial  Project  Schedule 

Acquisition  Category  I  Projects 


No  Impact  Some  Impact  Defining  Impact 


Acquisition  Category  II  Projects 


User's  Desires 
Expected  Dev  Funding 
Expected  Prod  Fundin 
Engineering  Requkem 
Testing  Requirements 
Technology  Developme 

Service  Planning 
Dep  on  Another  progr 
Serv  Leadership  Desi 
Support  Requirements 
Manufacturing  Proc  D 


No  Impact  Some  Impact  Defining  Impact 

Figure  8-17:  Factors  Affecting  Length  of  Initial  Project  Schedule,  by  Acquisition  Category  and 
Program  Size  (Program  Office  and  Pentagon  Surveys;  Number  of  Projects:  ACAT I  =  60, 
ACAT II  =  31,  ACAT  III  =  88) 
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Factors  Affecting  the  Length  of  the  Initial  Project  Schedule 


User's  Desires 
Expected  Prod  Fundln 
Expected  Dev  Funding 
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Service  Planning 
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Dep  on  Another  pcogr 
Technology  Devetopme 
Manufacturing  Proc  D 
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Figure  8-18:  Factors  Affecting  Length  of  a  Project’s  Initial  Schedule  by  Type  of  System 
(Program  Office  and  Pentagon  Surveys;  Number  of  Projects:  Aircraft  =  74,  Space  Systems 
=  8,  Electronic  Systems  =  21) 


No  Impact  Some  Impact  Defining  Impact 


Spacecraft  and  Launch  Systems 


No  Impact  Some  Impact  Defining  Impact 


Electronic  Systems  (Non- Aircraft) 


No  Impact  Some  Impact  Defining  Impact 
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Missile  and  Munitions  Systems 

User's  Desires 
Expected  Dev  Fencing 
Expected  Prod  Fundn 
Serv  Leodershlp  Desl 
Technology  Developme 
Testtno  Requirements 
Engineering  Requlrem 

Service  Planning 
Manufacturing  Proc  D 
Dep  on  Another  progr 
Support  Requirements 


No  Impact  Some  Impact  Defining  Impact 
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Engineering  Requlrem 
Dep  on  Another  progr 
Technology  Developme 
Testing  Requirements 
Expected  Prod  Fundln 
Serv  leadership  Desl 
Support  Requirements 
Manufacturing  Proc  D 
Service  Planning 


No  Impact  Some  Impact  Defining  Impact 
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Service  Planning 
Testing  Requirements 
Dep  on  Another  progr 
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No  Impact  Some  Impact  Defining  Impact 

Figure8-19:  Factors  Affecting  Length  of  a  Project’s  Initial  Schedule  by  Type  of  System 
(Program  Office  and  Pentagon  Surveys;  Number  of  Projects:  Missiles  and  Munitions  =  24, 
Software-Dominated  =  26,  Other  =  24) 


“Other”  Systems 
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Factors  Affecting  the  Length  of  the  Initial  Project  Schedule 


Expected  Dev  Funding 
Technology  Developme 
User's  Desires 
Engineering  Require m 
Serv  Leadership  Do  si 

Service  Planning 
Expected  Prod  F undin 
Testing  Requirements 
Manufacturing  Proc  D 
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Support  Requirements 


No  Impact  Some  Impact  Defining  Impact 
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Figure  8-20:  Factors  Affecting  Length  of  a  Project’s  Initial  Schedule  by  Technological 
Advance  of  the  Product  (Program  Office  and  Pentagon  Surveys;  Number  of  Projects: 
Revolutionary  =  19,  New  Generation  =  63,  Incremental  =  96) 


Incremental  Improvement  Projects 
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F.  Factors  Determining  a  Project’s  Initial  Funding  Profile 

Because  of  the  influence  of  expected  development  and  production  funding,  the  Pentagon 
survey  asked  respondents  to  rate  the  factors  affecting  a  project’s  initial  funding  profile.  The 
responses  identified  the  Pentagon’s  planning,  programming,  and  budgeting  system  (PPBS)  as  the 
primary  factor— rated  as  having  more  impact  nearly  twice  as  often  as  the  next  highest-rated 
factor,  -the  funding  profile  proposed  by  the  Program  Office.  All  other  factors  rated  consistently 
and  considerably  lower,  as  shown  in  Figure  8-23. 
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Figure  8-21:  Average  Responses  to  Factors  Influencing  a  Project’s  Initial  Funding  Profile 
(Pentagon  Survey;  Number  of  Projects  =  54) 


Another  observation  is  that  information-based  factors,  such  as  contractor-proposed 
funding  profiles,  historically  similar  programs,  bottom-up  or  task-based  development  of  funding 
profiles,  and  commercially  comparable  programs,  average  near  the  bottom  in  impact  on  initial 
funding  profile  at  the  Pentagon  level.  It  is  possible  that  these  factors  are  not  explicitly 
considered  at  the  Pentagon  level,  being  more  important  in  the  Program  Office  funding  profile. 
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However,  the  PPBS  system  appears  to  overshadow  even  the  Program  Office’s  proposed  funding 
profile  in  its  impact  on  initial  funding  profiles. 

Table  8-15  shows  the  number  of  projects  rating  the  PPBS  process  as  having  a  larger  or 
smaller  impact  than  other  factors  on  the  initial  funding  profile.  Pentagon  responses  rated  the 
PPBS  process  as  having  over  8  times  the  impact  of  the  user’s  proposed  funding  profile.  The  large 
disparity  between  the  impact  of  the  Pentagon  PPBS  process  and  the  user’s  proposed  funding 
profile  indicates  that  funding-related  decisions  are  made  primarily  at  the  Pentagon,  not  at  the 
major  commands. 


Programming,  Planning,  and  Budgeting  System  vs.  Other  Factors 


The  relative  impact  of  the  PPBS 
process  vs.  other  factors  in 
determining  a  project’s  initial  funding 
profile 

Other  Factors 

#  Projects  Rating 
the  Impact  of  the 
PPBS  Process 
Higher 

#  Projects  Rating 
Other  Factors 
Higher 

#  Rating 
Them 
Equal 

SPO  Proposed  Funding  Profile 

23 

12 

18 

Expected  Development  Funding 

23 

4 

26 

Expected  Service  Budget 

28 

5 

21 

Expert  Judgment 

31 

7 

16 

Expected  Production  Funding 

26 

5 

21 

Contractor  Proposed  Funding  Profile 

31 

9 

13 

Historical  Similar  Program 

35 

5 

14 

Bottoms  Up  Cost  Analysis 

38 

4 

12 

User  Proposed  Funding  Profile 

34 

4 

18 

Parametric  Modeling 

39 

5 

10 

Comparable  Commercial  development 

40 

4 

10 

Table  8-12:  Number  of  Projects  Reporting  the  Impact  of  the  Programming,  Planning,  and 
Budgeting  System  as  Larger,  Smaller,  or  Equal  to  Other  Factors  in  Determining  the  Initial 
Funding  Profile 
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G.  Schedule  Information  Used  to  Develop  a  Project’s  Initial  Schedule 

The  Program  Office  survey  asked  respondents  to  indicate  the  source  of  the  information 
they  used  to  develop  their  initial  project  schedule.  (The  list  of  potential  sources  was  based  on 
interviews  and  related  literature  reviews.)  The  results  indicate  that  the  primary  sources  of 
information  used  by  the  Program  Offices  were  expected  development  funding  and  “expert 
judgment”  based  on  the  participants’  experience.  Other  factors  related  to  development-related 
requirements,  such  as  bottom-up  schedule  development,  historically  similar  programs,  and 
parametric  modeling  rated  significantly  lower,  on  average,  and  on  a  project-by-project  and 
factor-by-factor  comparison.  Comparable  commercial  development  efforts  rated  significantly 
lower  than  all  other  sources  of  information. 

Follow-up  discussions  with  program  managers  and  other  individuals  revealed  that  the 
judgment  appeared  to  rest  on  an  individual’s  or  group’s  previous  experiences.  Program 
managers  most  often  described  “expert  judgment”  as  a  best  guess. 
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Figure  8-22:  Average  Responses  Regarding  Sources  of  Information  Used  To  Determine  a 
Project’s  Initial  Schedule  (Program  Office  Survey;  Number  of  Projects  =  130) 
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Relative  use  of  expected  development 
funding  versus  other  sources  of 
information  in  developing  a  project’s 
initial  schedule 

Other  Sources  of  Information 

#  Projects  Rating 
Expected 
Development 
Funding  Higher 

#  Projects  Rating 
Other  Tools  or 
Sources  of 
Information  Higher 

#  Rating 
Them 
Equal 

Expert  Judgment 

51 

41 

36 

Contractor  Proposal 

62 

29 

37 

Historically  Similar  Project 

64 

30 

34 

Bottoms  Up  Schedule  Development 

67 

19 

42 

Parametric  Modeling  or  Estimation 

85 

15 

24 

Commercial  Comparable  Development 

96 

12 

18 

Table  8-13:  Number  of  Projects  Reporting  Expected  Funding  as  Greater,  Lesser,  or  Equal 
to  Other  Information  Sources  in  Determining  a  Project’s  Initial  Schedule  (Program  Office 
Survey;  Number  of  Projects  =  130) 

Few  project  managers  reported  using  comparable  commercial  development  efforts  as  a 
source  of  information  in  establishing  initial  development  schedules.  In  fact,  a  majority  reported 
that  they  do  not  consider  such  experiences  in  developing  project  schedules.  These  results 
contrast  sharply  with  the  number  of  respondents  (49  percent)  who  attest  to  potential  commercial 
markets  for  key  technologies  in  their  projects. 


Not  Used  Occasionally  Used  Extensively  Used 


Comparable  Commercial 


Figure  8-23:  Reported  Use  of  Comparable  Commercial  Development  Efforts  in 
Establishing  Initial  the  Project  Schedule  (Program  Office  and  Pentagon  Surveys;  Number 
of  Projects  =  178) 
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H.  Use  of  Scheduling  Tools  to  Develop  a  Project’s  Initial  Schedule 

The  Program  Office  Survey  asked  project  managers  how  extensively  they  used  different 
scheduling  tools  to  develop  the  initial  schedule.  Five  potential  scheduling  tools  had  been 
identified  through  interviews  and  a  literature  search.  These  included  the  standard  Gantt  chart  and 
Milestone  charts  as  well  as  the  more  sophisticated  critical  path  management  and  PERT  systems, 
and  product  center-based  scheduling  models.  An  “other”  response  was  included  in  the  event  that 
a  key  tool  was  missed. 

Project  managers  reported  that  the  primary  tool  they  used  to  develop  the  initial  schedule 
was  the  relatively  unsophisticated  Milestone  chart  (86  of  126  projects  used  such  charts 
extensively).  Gantt  charts  were  the  second  most-used  scheduling  tool,  employed  in  58  projects. 
The  more  advanced  tools  were  used  significantly  less  frequently:  40  projects  used  critical  path 
management  extensively,  and  only  18  used  PERT  extensively.  Just  8  projects  indicated  that  they 
used  other  tools  extensively — with  Microsoft  Project  being  most  often  cited.  Only  4  projects 
used  center-based  schedule  models  extensively. 


Not  Used  Occasionally  Used  Extensively  Used 


Mean  +/- 1  SE 

Figure  8-24:  Schedule  Tools  Used  by  Program  Offices  to  Develop  a  Project’s  Initial 
Schedule  (Program  Office  Survey;  Number  of  Projects  =  126) 
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Other  Factors 

#  Projects 
Using 
Milestone 
Charts 

#  Projects 
Rating  Other 
Tools  Higher 

#  Rating 
Them 
Equal 

Gantt 

44 

19 

55 

Critical  Path  Management 

65 

12 

40 

PERT 

81 

9 

20 

Other 

38 

3 

8 

Center  Based  Model 

96 

1 

10 

Table  8-14:  Projects  Reporting  Use  of  Milestone  Charts  More  or  Less  Often  Than  Other 
Scheduling  Tools  in  Developing  the  Initial  Schedule  (Program  Office  Survey;  Number  of 
Projects  =  126) 


Scheduling  Tools 

Not 

Used 

Occasionally 

Used 

Extensively 

Used 

Numerical  rating 

1 

2 

3 

4 

5 

6 

7 

Milestone 

3 

1 

3 

15 

18 

47 

39 

Gantt 

13 

8 

5 

13 

24 

30 

28 

13 

11 

7 

27 

21 

25 

15 

PERT 

32 

13 

16 

22 

12 

10 

8 

Other 

37 

1 

1 

1 

1 

2 

6 

Center-Based  Model 

68 

15 

5 

9 

6 

3 

1 

Table  8-15:  Distribution  of  Responses  Regarding  Use  of  Various  Schedule  Tools  to 
Develop  a  Project’s  Initial  Schedule  (Program  Office  Survey;  Number  of  Projects  = 
126) 
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Summary  on  Development  of  a  Project’s  Initial  Schedule . 

A  project’s  initial  schedule  is  not  determined  by  technology,  engineering,  or 
manufacturing  requirements  but  instead  primarily  by  funding-related  constraints.  Respondents 
reported  that  users  wanted  80  percent  of  current  projects  as  soon  as  possible,  and  that  70  percent 
of  projects  are  intended  to  remedy  existing  deficiencies.  But  despite  this  reported  need  and  sense 
of  high  project  priority,  project  managers  reported  short  schedules  as  the  lowest  project  priority. 

The  starting  date  for  a  project  is  most  often  determined  by  the  user’s  or  leadership’s 
desires.  Secondary  items  in  determining  when  to  start  a  project  are  funding  availability  and 
service  planning.  Product  development-related  aspects,  including  technology  development, 
engineering  requirements,  and  manufacturing  processes,  do  not  play  a  significant  role  in 
determining  when  to  start  a  project. 

Program  Offices  primarily  control  the  process  used  to  develop  the  length  of  a  project’s 
initial  schedule,  with  users,  the  service  acquisition  community,  and  potential  contractors  exerting 
less  influence.  Funding-based  constraints,  including  the  user’s  desires  as  limited  by  funding,  and 
expected  development  and  production  funding,  are  most  important  in  establishing  the  length  of  a 
project’s  initial  schedule.  Product  development  requirements,  including  engineering 
requirements,  technology  development,  and  manufacturing  processes,  rank  well  below  funding- 
related  constraints.  This  finding  is  consistent  across  projects  of  all  sizes,  types,  and  levels  of 
technical  advance.  Schedules  for  80  percent  of  current  projects  are  limited  by  funding  and  not  by 
technical  or  engineering  constraints. 

The  primary  sources  of  information  used  to  develop  initial  project  schedules  are  expected 
development  funding  and  expert  judgment.  More  analytically  based  information  as  parametric 
modeling,  historical  similar  programs,  bottom-up  schedule  development,  and  comparable 
commercial  programs  have  significantly  less  influence  on  project  schedules.  The  primary  tools 
used  to  develop  initial  project  schedules  are  the  basic  Milestone  and  Gantt  charts.  The  more 
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sophisticated  Critical  Path  Management  and  PERT  are  rarely  used  extensively,  though  many 
programs  report  using  those  tools  occasionally. 

The  initial  project  schedule  sets  the  stage  for  further  acquisition  planning  and  approval, 
which  in  turn  leads  to  the  contracting  phase.  The  impact  of  a  project’s  initial  schedule  on 
contractors’  proposed  schedules  and  actual  contracted  schedules  is  addressed  in  the  next  chapter. 


Chapter  9 


The  Impact  of  a  Project’s  Initial  Schedule  on 
the  Contracting  Process 


Defense  contractors  accomplish  nearly  all  Air  Force  development  projects.  The  contract 
established  between  the  Program  Office  and  the  company  describes  what  needs  to  be  done  and 
when  it  needs  to  be  completed.  This  contracted  schedule  is  developed  from  the  government’s 
initial  schedule  and  the  contractor’s  proposed  schedule,  which  is  in  turn  affected  by  schedule- 
related  incentives  from  the  Program  Office.  As  will  be  shown  in  Chapters  10  and  1 1,  the 
contracted  schedule  plays  the  central  role  in  determining  the  time  it  takes  to  develop  a  project. 
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Figure  9-1:  Development  of  the  Contract  Schedule  During  a  Project’s  Contracting  Phase 
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A.  The  Expected  Schedule  Included  in  the  Request  for  Proposals 

A  significant  number  of  projects  publish  an  expected  schedule  as  part  of  the  request  for 
proposal  (RFP).  To  determine  how  often  that  occurs,  the  Pentagon  survey  asked  if  the  project 
had,  through  its  RFP  or  other  means,  specified  an  expected  schedule  to  potential  contractors.  Of 
those  respondents,  80  percent  said  that  the  Program  Office  had  established  an  expected  schedule 
for  the  contractors. 


“  Did  the  Government,  through  its  RFP  or  other  means, 
specify  an  expected  project  schedule  to  the  contractors?” 


Specified 

Expected 

Schedule 


Schedule 

Not 

Specified 


Figure  9-2:  Percentage  of  Projects  Reporting  a  Project  Schedule  Specified  to  Contractors. 
Pentagon  Survey;  Number  of  Projects  =  48 

These  results  are  consistent  with  those  in  a  master’s  thesis  completed  under  the  purview 
of  the  Air  Force  Institute  of  Technology.  Richard  Hazeldean  and  John  Topfler  found  that  20  of 
25  development  contracts  they  studied  at  Wright-Patterson  Air  Force  Base  were  “pre-scheduled,” 
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and  had  included  an  expected  schedule  in  the  request  for  proposal.’7 

Before  examining  the  effect  of  the  government’s  projected  schedule,  it  is  useful  to  look  at 
contractors’  view  of  their  ability  to  influence  project  planning  prior  to  release  of  the  RFP. 

B.  Contractor’s  Ability  to  Influence  the  Request  for  Proposal 

The  contractor  survey  asked  respondents  to  rate  their  company’s  influence  on  various 
aspects  of  the  project  prior  to  release  of  the  RFP.  The  contractors  reported  having  the  most 
influence  on  the  project  concept.  They  reported  significantly  less  ability  to  influence  the  system 
performance  requirements,  and  even  less  ability  to  influence  project  schedules.  The  contractor’s 
influence  on  trade-offs  among  performance,  cost,  and  schedule;  program  funding  levels;  and  the 
project’s  acquisition  strategy  fell  in  the  lowest  statistical  group. 
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Figure  9-3:  Contractors’  Reported  Ability  to  Influence  a  Government  RFP  Prior  to  Its 
Release  (Number  of  Projects  =  99) 


97  Richard  Hazeldean  and  John  Topfer.  Contracting  for  Schedule  Perfromance.  Air  Force  Institute  of  Technology. 
AFIT/GSM/93S-7  Wright  Pattterson  Air  Force  Base,  OH.  pg.  4-10. 
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On  average,  contractors  report  having  less  than  “some  impact”  on  the 
government’s  schedule  development  prior  to  RFP  release.  The  Program  Office  survey 
indicated  that  contractors  were  the  third  most  important  organization  affecting  project 
schedule  development.  Results  of  the  Pentagon  survey  ranked  contractors  fifth  in 
influence  on  development  schedules. 

The  type  of  contra  ;t  award  appears  to  have  a  significant  impact  on  contractors’ 
ability  to  influence  a  program.  Contractors  with  competitively  awarded  contracts  appear 
to  have  dramatically  less  influence  on  a  project’s  schedule,  as  well  as  on  the  projected 
budget  profile  and  acquisition  strategies.  No  statistical  differences  were  noted  in 
contractors’  abilities  to  influence  performance  requirements,  product  concept,  or  the 
trade-off  among  cost,  schedule,  and  performance,  based  on  type  of  award. 
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Figure  9-4:  Impact  of  Competition  on  a  Contractor’s  Ability  to  Influence  Various  Aspects 
of  a  Project  (Statistical  Differences  Determined  Using  the  Wilcoxon  Signed-Rank  Test) 
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C.  Contractor  Schedule  Inputs 

If  contractors  have  little  ability  to  influence  the  government’s  expected  schedule 
prior  to  RFP  release,  what  impact  does  that  schedule  have  on  the  contractor’s  proposed 
schedule?  In  developing  a  proposed  schedule,  contractors  consider  many  factors.  The 
contractor  survey  asked  respondents  to  rank  five  such  factors  from  “no  impact”  to  “sole 
determinant.”  The  customer’s  desired  schedule  was  reported  to  have  the  largest  impact 
on  contractors’  proposed  schedules  by  a  wide  margin.  The  factors  associated  with  the 
actual  development  requirements,  including  expert  judgment,  bottom-up  schedule 
development,  historically  similar  programs,  and  comparable  commercial  development, 
were  all  rated  significantly  lower  than  the  customer’s  desired  schedule.  Thirty-two 
percent  of  respondents  indicated  that  the  customer’s  desired  schedule  was  the  “sole 
determinant,”  and  another  34  percent  selected  the  next-highest  rating.  None  of  the  other 
factors,  including  “expected  funding  availability,”  “expert  judgment/bottom-up  schedule 
development,”  and  “historical  similar  programs,”  averaged  above  “moderate  impact.” 


Mean  +/- 1  SE 

Figure  9-5:  Impact  of  Various  Factors  on  a  Contractor’s  Proposed  Schedule  (Contractor 
Survey;  Number  of  Projects  =  96) 


226  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Customer’s  Desired  Schedule 


Figure  9-6:  Impact  of  Customer’s  Desired  Schedule  on  Contractor-Proposed  Schedule 
(Contractor  Survey;  Number  of  Projects  =  96) 


Project  characteristics  such  as  product  type  (aircraft,  spacecraft,  electronic  system,  or 
munitions),  product  size  (ACAT  level),  and  acquisition  phase  did  not  have  a  significant  impact 
on  the  reported  contractor  schedule  inputs.  The  type  of  contract  selection  (sole  source  or 
competitive  source)  also  did  not  have  a  large  impact  on  the  contractor’s  schedule  development. 
The  customer’s  desired  schedule  was  rated  slightly  higher  for  competitive  procurements  than  for 
sole-source  awards,  but  the  difference  was  statistically  significant  at  only  the  0.15  confidence 
level.  Contractors  on  both  types  of  contracts  reported  primarily  using  the  customer’s  desired 
schedule. 
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No  Impact  Moderate  Impact  Sole  Determinant 


Figure  9-7:  Effect  of  Competition  on  Contractor  Schedule  Inputs  by  Type  of  Contract 
Award  (Contractor  Survey;  Number  of  Projects  =  96) 


Comparable  commercial  development  projects  were  rated  as  having  the  lowest  impact  on 
the  contractor’s  proposed  schedule,  and  appeared  not  to  play  any  significant  role  in  most 
projects:.  61  of  96  reported  “no  impact.”  Overall,  81  percent  of  respondents  chose  one  of  the  two 
lowest  categories.  Comparable  commercial  development  efforts  were  rated  higher  than  expert 
judgment  in  5  of  95  cases,  higher  than  historically  similar  programs  in  2  cases,  and  higher  than 
expected  funding  in  2  cases. 
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Figure  9-8:  Impact  of  Comparable  Commercial  Development  on  a  Contractor’s  Proposed 
Schedule  (Contractor  Survey;  Number  of  Projects  =  96) 


The  very  low  impact  of  comparable  commercial  efforts  would  imply  that  such 
efforts  are  not  relevant  to  military  projects.  However,  responses  to  other  questions  indicate  that 
it  is  not  the  technology  of  military  projects  that  separates  them  from  commercial  projects. 

To  determine  the  relevance  of  commercial  development  efforts,  both  the  contractor  and 
Pentagon  surveys  asked  if  the  key  enabling  technologies  used  in  the  projects  had  existing 
commercial  markets.  Among  162  projects,  68  percent  reported  at  least  “some  commercial 
markets”  for  key  enabling  technologies  used  in  the  project.  In  a  separate  question,  the  Program 
Office  survey  asked  whether  technologies  used  in  the  project  had  more  demanding  requirements 
than  commercial  technologies,  and  whether  the  system  should  therefore  take  longer  to  develop 
and  field.  Forty-three  percent  did  not  agree,  indicating  that  they  believed  projects  they  were 
working  on  should  not  take  any  longer.  An  additional  29  percent  indicated  that  they  “somewhat 
agreed,”  while  29  percent  indicated  that  they  “strongly  agreed”  that  the  projects  should  take 
longer  to  develop. 
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These  results  indicate  that  in  many  cases  commercial  development  efforts  use  similar 
technologies.  The  results  also  indicate  that  factors  other  than  technology  underlie  the  low  impact 
of  comparable  commercial  projects  on  schedules  for  military  projects. 

D.  Contractors’  Incentives  and  Source  Selection  Criteria 

With  contractors  reporting  that  the  customer’s  desired  schedule  is  the  dominate  influence 
on  their  proposed  schedule,  it  was  important  to  determine  the  schedule-related  incentives  that 
contractors  experience.  The  surveys  thus  attempted  to  determine  the  importance  of  the 
development  schedule  in  selection  of  a  contractor,  and  the  incentives  for  contractors  to  propose 
alternative  schedules.  During  proposal  development,  a  company’s  primary  objective  is  to  be 
selected  for  the  contract.  Failure  likely  means  exclusion  not  only  from  development  but  also 
from  production  and  long-term  support  activities. 

The  Program  Office  and  Pentagon  surveys  queried  respondents  on  the  importance  of  the 
development  schedule  in  choice  of  contractor.  Among  program  managers,  78  percent  listed 
development  time  as  “somewhat  important”  or  “not  important”  as  a  source  selection  criterion. 
Fully  30  percent  identified  development  time  as  “not  important,”  while  only  22  percent  identified 
it  as  “very  important.” 
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Figure  9-9:  Importance  of  Development  Time  as  a  Source  Selection  Criterion  (Program 
Office  and  Pentagon  Surveys;  Number  of  Projects  =  151) 


To  determine  the  impact  of  source  selection  criteria  on  contractors  as  they  develop  their 
proposals,  the  Contractor  Survey  asked  respondents  to  indicate  their  net  incentive  to  bid  a 
schedule  different  from  that  proposed  by  the  government.  Sixty  percent  responded  that  they  had 
“no  incentive”  to  bid  a  different  schedule,  while  15  percent  indicated  that  they  had  a  negative 
incentive.  Only  25  percent  indicated  a  positive  incentive,  and  just  6  programs  indicated  a  strong 
incentive  to  bid  a  different  schedule.  The  responses  reveal  that  contractors  feel  little  incentive  to 
bid  schedules  other  than  what  Program  Offices  expect. 
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Figure  9-10:  Contractor’s  Incentive  to  Bid  a  Schedule  Different  from  the  Expected 
Schedule  (Contractor  Survey;  Number  of  Projects  =  97) 

There  may  be  several  explanations  for  the  fact  that  contractors  do  not  view  development 
time  as  an  important  selection  criterion,  including  the  lack  of  overall  priority  given  to 
development  schedules,  as  shown  in  the  last  chapter.  However,  experience  in  the  commercial 
market  might  suggest  that  competition  would  increase  contractors’  incentives  to  reduce  time  to 
market  and  bid  a  shorter  schedule.  But  the  survey  results  indicate  that  the  opposite  is  true: 
contractors  involved  in  competitive  source  selections  report  that  they  have  less  incentive  to 
submit  a  proposal  schedule  different  from  the  government’s  estimate.  In  both  sole-source  and 
competitive  selection,  there  appears  to  be  little  incentive  to  bid  a  schedule  different  from  what 
Program  Offices  expect.  Competition  based  on  time  to  market  does  not  generally  appear  to 
occur  in  defense  development  projects. 
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Figure  9-11:  Distribution  of  Reported  Contractor  Incentive  for  Bidding  Different 
Schedules,  for  Sole-Source  and  Competitive  Awards  (Contractor  Survey  ;  N  =  96) 


During  follow-up  interviews,  several  contractors  expressed  surprise  that 
incentives  to  bid  a  different  schedule  were  even  this  high:  they  believe  there  is  generally  a 
strong  disincentive  to  bid  different  schedules.  The  reason  they  provided  is  that  the 
Program  Offices  believe  they  understand  how  long  it  will  take  to  develop  the  project,  and 
if  contractors  bid  different  schedules  they  are  seen  as  higher-risk  or  non-responsive.  This 
attitude  is  seen  in  companies’  guidance  to  proposal  development  teams  outlined  in 
Chapter  3.  Such  guidance  emphasizes  that  telling  Program  Offices  they  are  wrong  “is  not 
a  winning  strategy.” 

Contractors  often  have  the  option  of  submitting  an  alternative  proposal  in  response  to  an 
RFP.  Several  interviews  with  Government  program  managers  at  the  Aeronautical  Systems 
Center  and  the  Electronic  Systems  Center,  however,  revealed  that  Program  Offices  rarely 
consider  alternative  proposals.  Interviewees  believed  that  submitting  an  alternative  proposal  split 
the  contractor’s  proposal  preparation  effort  and  resulted  in  two  lower-quality  proposals,  each 
with  a  smaller  chance  of  selection.  Contractors  also  stated  that  submitting  two  proposals  might 
indicate  that  the  company  could  not  make  up  its  mind,  revealing  a  lack  of  ability  to  make  hard 
decisions.  They  stated  that  the  typical  approach  is  to  submit  the  proposal  that  stands  the  highest 
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chance  of  being  selected,  and  then  to  discredit  the  other  approach  in  case  a  competitor  has 
selected  that  approach.  Such  “ghosting  “  of  non-selected  approaches  is  also  documented  in  one 
company’s  proposal  development  guide  Interviewees  at  the  Aeronautical  System  Center 
stated  that  an  RFP  often  forbids  alternative  proposals,  and  that  when  not  forbidden  they  are  rarely 
encouraged.  They  stated  that  proposals  not  meeting  the  expectations  of  the  Program  Offices  are 
viewed  with  suspicion  and  generally  automatically  perceived  to  have  higher  risk.  They  also 
stated  that  for  a  company’s  alternate  proposal  to  be  considered,  its  primary  proposal  would  have 
to  be  in  the  competitive  range  and  stand  a  good  chance  of  being  selected. 


E.  Resulting  Contractor  Proposals 

Any  effect  of  schedule  incentives  would  appear  in  contractors’  proposals.  The  Contractor 
survey  asked  respondents  to  quantify  the  difference  between  their  company’s  proposed  project 
schedule  and  the  Program  Office’s  expected  schedule  outlined  in  the  RFP.  Of  83  respondents, 
62or  75  percent  said  their  proposed  schedules  matched  the  length  of  the  Program  Office’s 
planned  schedule  exactly. 
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Figure  9-12:  Difference  Between  Winning  Contractor’s  Proposed  Schedule  and  the 
Government’s  Expected  Schedule  (Contractor  Survey;  N  =  83) 


The  responding  contractors  do  not  represent  all  the  companies  that  bid  on  the  projects, 
only  those  that  won  contracts.  To  examine  the  larger  set  of  all  bidding  companies,  the  Program 
Office  survey  asked  respondents  to  identify  all  the  proposals  received.  Of  the  282  proposals 
reported,  only  32  were  shorter  than  the  government’s  planned  schedule  by  more  than  5  percent. 
Some  194  were  within  5  percent,  and  56  were  longer  than  the  government  schedule  by  more  than 
5  percent. 
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Figure  9-13:  Proposals  Received  by  Program  Offices  Compared  with  Their  Expected 
Schedule  (Program  Office  Survey;  Number  of  Projects  =  83) 

The  Pentagon  survey  found  similar  results.  Of  the  30  Pentagon  respondents  who  had 
completed  the  source  selection  process,  all  but  6  selected  contractors  who  bid  exactly  the 
schedules  expected  by  the  Program  Office.  Four  of  the  6  winning  contractors  who  bid  different 
schedules  bid  shorter  schedules,  while  2  bid  longer  schedules. 

The  effect  of  competition  on  contractors’  proposed  schedule  is  opposite  to  would  have 
been  expected  from  commercial  experience.  On  average,  contractors  participating  in  a 
competitive  selection  process  proposed  project  schedules  4.5  percent  longer  than  the  Program 
Office  estimate.  Schedules  from  contractors  operating  in  a  sole-source  environment  were  3 
percent  shorter,  on  average,  than  the  Program  Office  estimate.  However,  while  this  difference  is 
significant  at  the  92  percent  level  using  the  independent  samples  t-test,  caution  should  be  taken 
as  the  number  of  projects  bidding  different  schedules  is  small.  A  vast  majority  of  both 
competitive  and  sole-source  proposals  are  no  different  from  the  Program  Office’s  expected 
schedule. 
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Figure  9-14:  Percentage  Difference  between  the  Winning  Contractor’s  Proposed  Schedule 
and  the  Program  Office’s  Expected  Schedule  (Pentagon  Survey;  Number  of  Projects  =  32) 
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Figure  9-15:  Percent  Difference  between  the  Program  Office’s  Expected  Schedule  and  the 
Contractor’s  Proposed  Schedule  for  Competitive  and  Sole-Source  Awards  (Contractor 
Survey;  Number  of  Competitive  Contracts  =  44;  Number  of  Sole-Source  Contracts  =  39 
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Summary  of  Contracting-Phase  Schedule  Results 

During  the  contracting  phase,  the  Program  Office  selects  plans  for  the  development 
project  as  well  as  a  contractor  to  execute  them.  This  is  done  through  a  request  for  proposal  on 
which  contractors  bid.  The  contracted  schedule  between  the  Program  Office  and  the  winning 
company  is  very  important,  as  it  that  details  what  the  contractor  is  to  accomplish  and  when. 

Development  of  the  contract  schedule  is  driven  primarily  by  the  project’s  initial  planned 
schedule  and  not  by  other  factors.  Overall  80  percent  of  projects  specify  an  expected  schedule  to 
contractors.  A  contractor’s  primary  consideration  in  proposing  a  project  schedule  is  the  Program 
Office’s  desired  schedule.  The  company’s  development  capabilities  are  given  much  less 
consideration.  When  developing  a  proposed  schedule,  contractors  report  no  incentives  to  bid 
anything  other  than  what  the  government  has  requested.  Program  Office  and  Pentagon 
respondents  report  that  short  schedules  are  not  an  important  source  selection  criterion,  and  do  not 
give  a  contractor  any  advantage  in  that  process.  Bidding  a  schedule  different  from  what  is 
expected  increases  the  perceived  risk  to  the  government  and  the  contractors.  Because  companies 
that  bid  a  different  schedule  are  considered  “non-responsive”  to  the  RFP,  they  are  less  likely  to 
be  selected,  are.  Thus  industry  parrots  back  government-estimated  schedules  with  few  changes. 
From  these  contractor  proposals  the  Program  Offices  select  the  winners  and  award  the  contracts. 

As  was  shown  in  the  previous  chapter,  a  project’s  initial  schedule  is  determined  primarily 
by  the  available  funding.  The  overall  result  of  the  planning  and  contracting  phases  are  that 
contract  schedules  are  based  primarily  on  the  Program  Office’s  interpretation  of  a  project’s 
funding  constraints,  not  on  its  development-related  requirements  or  the  contractor’s  product 
development  capabilities. 
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Awarding  the  development  contract  marks  the  end  of  the  contracting  phase  and  the 
beginning  of  the  development  phase.  The  latter  phase  ends  with  the  delivery  of  the  first 
production  item,  marking  the  point  when  the  vast  majority  of  development-related  activities  are 
complete.  Although  the  contracted  schedule  plays  a  significant  role  in  setting  product 
development  times,  it  does  not  necessarily  determine  the  amount  of  time  it  will  actually  take  to 
develop  a  new  system.  Actual  development  times  can  be  longer  or  shorter  depending  on  many 
factors  ranging  from  technical  problems,  to  funding  constraints,  to  changing  Air  Force 
objectives.  This  chapter  looks  at  the  incentives  for  different  organizations  associated  with 
project  schedules,  from  the  time  of  contract  award  until  delivery  of  the  first  production  item. 

To  understand  these  incentives  three  surveys — of  the  Pentagon,  Program  Offices,  and 
contractors — asked  respondents  to  report  the  incentives  to  meet  or  exceed  various  project 
objectives  they  had  experienced..  A  series  of  follow-up  interviews  with  representatives  from 
each  group  helped  interpret  and  explain  the  results. 
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Figure  10-1:  Project  Schedules  in  the  Development  Phase 


A.  Users’  Incentives  for  Reducing  Cycle  Time 

Because  the  surveys  did  not  target  the  user  community  directly,  Pentagon  respondents,  as 
the  closest  available  surrogates,  were  asked  a  number  of  questions  about  users’  incentives. 
Program  element  monitors  and  action  officers  are  the  spokespeople  for  users  and  projects  within 
the  Pentagon,  and  as  such  are  expected  to  be  aware  of  users’  objectives  and  priorities.  The 
responses  indicate  that  users  have  a  large  incentive  to  meet  a  project’s  planned  cost,  schedule, 
and  performance  goals.  The  responses  also  indicated  that  many  users  have  a  significant  incentive 
to  exceed  project  goals  in  all  aspects  of  cost,  schedule,  performance,  and  reliability.  The 
incentives  for  increasing  product  reliability,  reducing  project  schedules,  cutting  total  project  cost, 
enhancing  technical  performance,  and  reducing  unit  costs  were  all  positive.  The  reported 
differences  between  various  project  objectives  were  not  statistically  significant. 

Overall,  Pentagon  respondents  indicate  that  65  percent  of  users  have  a  positive  incentive 
to  shorten  a  project  schedule;  47  percent  indicated  that  the  users  had  a  significant  or  strong 
incentive  to  shorten  a  project’s  schedule.  Of  those  that  provided  a  description  of  the  schedule- 
related  incentives  for  users,  many  mentioned  the  need  for  enhanced  combat  capability  to  meet 
operational  objectives.  Others  mentioned  the  lower  operational  and  maintenance  costs  promised 
by  the  new  systems. 
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Figure  10-2:  User  Incentives  to  Meet  and  Exceed  Project  Objectives  as  Reported  by 
Pentagon  Survey  Respondents  (Number  of  Projects  =  60) 
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Figure  10-3:  Average  of  Users’  Incentives  to  Meet  or  Exceed  Project  Objectives  (Pentagon 
Survey;  Number  of  Projects  =  60) 
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Figure  10-4:  Users’  Incentives  to  Reduce  Development  Time  (Pentagon  Survey; 
Number  of  Projects  =  49) 


B.  Incentives  for  Reducing  Cycle  Time  within  the  Pentagon 

To  determine  the  incentives  within  the  Pentagon  to  meet  or  exceed  project  objectives,  the 
Pentagon  survey  asked  respondents  to  rate  various  project  objectives  as  viewed  by  their 
organization,  and  as  they  viewed  them  personally.  The  survey  also  asked  what  impact  meeting 
or  exceeding  project  objectives  had  on  their  personal  performance  rating  and  their  potential  for 
promotion. 

The  survey  asked  respondents  to  rank-order  four  project  objective  as  viewed  by  their 
organizations:  low  acquisition  cost,  low  operation  and  support  cost,  superior  technical 
performance,  and  a  short  schedule  to  operational  capability.  The  respondents  most  often  rated 
superior  performance  as  the  first  priority,  low  acquisition  cost  as  the  second  priority,  and  low 
operating  and  maintenance  costs  as  the  third  priority.  Short  schedule  was  most  often  rated  as  the 
fourth  of  four  objectives,  and  rated  significantly  lower  than  each  of  the  other  objectives.  A  total 
of  65  percent  of  respondents  listed  short  schedule  as  either  the  third  or  fourth  of  the  four  project 
objectives,  while  42  percent  rated  short  schedule  as  the  fourth  of  four  objectives. 
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Pentagon  respondents  were  also  asked  to  rank-order  the  four  objectives  as  they 
personally  viewed  them  to  determine  if  their  views  differed  from  those  of  their  organizations. 
Here  the  most  important  objective  was  reported  as  lowering  a  project’s  acquisition  cost.  Other 
objectives  such  as  improving  system  performance,  lowering  operating  costs,  or  shortening  the 
schedule  were  rated  significantly  lower.  Short  schedule  was  statistically  tied  with  low  operating 
costs  for  last  of  the  four  objectives. 

Given  the  low  priority  placed  on  shortening  schedules  compared  with  other  objectives,  it 
is  unlikely  that  program  element  monitors  or  Air  Staff  action  officers  would  push  for  reducing 
project  schedules.  They  would  be  more  likely  to  work  toward  achieving  either  what  they  view 
their  organization  wants  or  what  they  personally  view  as  most  important:  improving  performance 
or  reducing  acquisition  cost.  Shortening  the  schedule  is  not  generally  seen  as  an  important 
organizational  or  personal  priority. 


Figure  10-5:  Perceived  Project  Objectives  from  Pentagon  Respondents. 
(Number  of  Projects  =  60) 
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Figure  10-6:  Respondents’  Personal  View  of  the  Importance  of  Various  Objectives  to  the 
Development  Effort.  (Pentagon  Survey;  Number  of  Projects  =  60) 

The  Pentagon  survey  also  asked  respondents  to  indicate  how  not  meeting,  meeting,  or 
exceeding  their  project  objectives  affected  their  personal  performance  ratings  and  potential  for 
promotion.  The  most  frequent  responses  in  each  category  indicated  that  the  success  or  failure  of 
a  program  has  no  impact  on  respondents’  performance  rating  or  potential  for  promotion.  Overall 
85  percent  replied  that  exceeding  objectives  would  have  only  “some  impact”  or  less  on  their 
performance  ratings.  And  75  percent  stated  that  not  meeting  the  project’s  objective  would  have 
“some”  or  small  impact  on  their  rating  or  potential  for  promotion. 

In  follow-up  interviews,  several  program  element  monitors  reiterated  that  the  success  of  a 
project  had  little  effect  on  their  personal  performance  rating  or  potential  for  promotion,  and 
indicated  they  felt  that  it  should  not  affect  their  performance  rating.  Those  interviewed  felt  that 
since  they  had  little  control  over  the  success  of  a  program,  it  would  be  unfair  to  rate  them  by  the 
success  or  failure  of  the  program.  Some  program  element  monitors  seemed  somewhat  detached 
from  their  specific  programs,  indicating  that  they  had  to  support  what  was  good  for  the  entire  Air 
Force  and  could  not  just  push  for  their  specific  program.  Others  were  strong  advocates  and 
appeared  very  active  in  planning  and  executing  a  program.  According  to  regulations,  the 
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program  element  monitor  is  the  program’s  official  representative,  spokesperson,  and  advocate 
within  the  Pentagon.  Several  Pentagon-level  respondents  indicated  that  other  factors  such  as 
appearance,  hours  worked,  and  responsiveness  to  requests  for  information  were  more  important 
on  their  performance  ratings  than  program  success. 


Percent  ,  Percent  ,  Percent 
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Figure  10-7:  Pentagon  Respondents’  Perceived  Impact  of  Not  Meeting,  Meeting,  or 
Exceeding  Project  Goals  on  Personal  Performance  Ratings  and  Potential  for  Promotion 
(Number  of  Projects  =  58) 
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C.  Program  Office  Incentives  for  Reducing  Development  Time 

The  Program  Office  survey  queried  project  managers  about  incentives  to  meet  or  exceed 
project  objectives;  whether  short  schedules  were  viewed  as  an  important  objective;  and  the  effect 
of  exceeding  various  objectives  on  their  personal  performance  ratings.  The  Program  Office 
survey  also  asked  about  the  continuity  and  tenure  of  project  managers  and  project  staff.  To 
determine  incentives  transmitted  to  Program  Offices  frontforganizations  within  the  Pentagon,  the 
Pentagon  survey  also  asked  respondents  about  incentives  the  Program  Offices  had  to  exceed 
various  objectives.  Again  the  results  indicate  few  significant  incentives  for  reducing  project 
schedules  in  the  Program  Offices. 

The  Program  Office  Survey  asked  project  managers  to  rank-order  four  project  objectives: 
superior  technical  performance,  low  acquisition  cost,  low  operations  and  maintenance  costs,  and 
short  schedule  to  operational  capability.  The  largest  number  of  project  managers  indicated  that 
superior  performance  was  the  first  project  objective.  Low  acquisition  cost  was  rated  most  often 
as  second.  Low  operations  and  maintenance  costs  were  most  often  rated  as  third.  Shorter 
schedule  was  again  most  often  rated  as  the  fourth  of  four  project  objectives.  Shortened  schedule 
was  rated  significantly  lower  than  superior  performance  and  low  acquisition  cost,  and  roughly 
equivalent  to  low  maintenance  costs.  As  with  Pentagon  respondents,  project  managers  are  likely 
to  focus  on  what  they  see  as  the  first  or  second  project  objective.  Any  additional  effort  on  the 
part  of  project  managers  is  likely  to  be  aimed  at  better  performance  or  lower  acquisition  cost,  and 
not  at  shorter  cycle  times. 

Another  question  asked  government  project  managers  what  extent  a  short  acquisition 
cycle  was  an  important  part  projects  objectives.  As  shown  in  Figure  10-9,  the  distribution  of 
responses  was  spread  roughly  equally  from  not  important  to  very  important.  Manager  of  36 
projects,  or  25  percent,  indicated  that  short  schedules  were  not  important.  Managers  of  59 
projects,  or  41  percent,  said  short  schedules  were  “somewhat  important.”  Managers  of  only  49 
projects,  or  34  percent,  indicated  that  short  schedules  were  an  important  project  objective.  Thus 
a  substantial  number  (66  percent)  of  project  managers  do  not  see  a  short  schedule  as  a 
particularly  important  objective  for  their  projects,  even  when  not  competing  against  other  project 
objectives. 
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Figure  10-8:  Ranking  of  Various  Project  Objectives  by  Government  Project  Managers 
(Program  Office  Survey;  Number  of  Projects  =  147) 


Not  Important  Somewhat  Important  Very  Important 


Figure  10-9:  Number  of  Project  Managers  Reporting  Short  Acquisition  Cycle  as  Very 
Important,  Somewhat  Important,  or  Not  Important  (Program  Office  Survey;  Number  of 
Projects  =  144) 
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The  Pentagon  survey  asked  respondents  to  indicate  the  incentives  for  Program  Offices  to 
meet  or  exceed  various  project  objectives.  The  Pentagon  respondents  indicated  that  the  Program 
Offices  have  some  incentive  to  meet  project  objectives  but  very  little  incentive  to  exceed  them. 
The  incentives  for  exceeding  project  objectives  were  reported  to  be  significantly  lower  than  those 
reported  for  users. 

As  shown  in  Figures  10-10  and  10-1 1,  the  program  element  monitors  believe  tha|  the 
Program  Offices’  incentives  for  achieving  a  shortened  schedule  are  significantly  lower  than  for 
meeting  the  schedule.  Only  20  percent  of  projects  were  reported  with  significant  incentives  to 
shorten  schedule,  and  23  percent  with  a  slight  incentive.  Thus  58  percent  of  the  projects  were 
reported  to  have  either  no  incentive  (46  percent)  or  a  negative  incentive  (12  percent)  to  shorten 
the  schedule.  Program  Offices  are  seen  as  having  a  significantly  stronger  incentive  to  meet,  as 
opposed  to  exceed,  program  objectives,  with  80  percent  of  respondents  indicating  they  had  a 
positive  incentive  to  do  so. 


Meet  Cost,  Schedule, 
Performance  Goals 

Reduce  Unit  Cost 

Reduce  Total  Cost 

Exceed  Reliability 
Requirements 

Shorten  Schedule 

Exceed  Performance 
Requirements 
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Figure  10-10:  Pentagon  View  of  Program  Offices’  Incentives  for  Meeting  or  Exceeding 
Project  Objectives.  (Pentagon  Survey;  Number  of  Projects  =  59) 
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Figure  10-11:  Pentagon  View  of  Program  Offices’  Incentive  to  Reduce  Development  Time 
(Pentagon  Survey;  Number  of  Projects  =  59) 


The  Pentagon  survey  asked  respondents  to  indicate  the  form  or  type  of  incentive  for 
Program  Offices  to  achieve  or  exceed  objectives.  Very  few  responses  indicated  any  specific, 
structured,  or  formalized  incentive.  Many  mentioned  oversight  by  leaders  from  the  services  and 
DoD  as  a  motivating  factor.  Most  others  indicated  “doing  the  right  thing,”  individual  initiative, 
and  “patriotic  duty”  as  the  incentives.  Few  pointed  to  a  specific  organizational  or  individual 
financial,  personal  rating,  or  position-based  incentive  for  Program  Offices.  Many  indicated  that 
the  objectives  of  the  program  were  to  achieve  their  goals  and  not  to  exceed  them. 

To  measure  personal  incentives,  the  Program  Office  survey  asked  project  managers  rank- 
order  various  project  goals — completing  under  budget,  completing  ahead  of  schedule,  exceeding 
technical  performance  requirements,  or  exceeding  operability/  maintenance  requirements — by 
their  importance  to  their  performance  rating  and  potential  for  promotion.  Meeting  project  goals 
was  not  included  in  this  question  as  an  option.  Of  the  four  objectives,  finishing  below  cost  was 
rated  most  often  as  having  the  most  impact  on  project  managers’  performance  ratings. 

Delivering  a  project  ahead  of  schedule  was  most  often  rated  as  having  the  second-largest  impact. 
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As  illustrated  in  Figure  10-12,  exceeding  operability  and  maintenance  requirements  was  rated 
last  on  program  managers’  report  by  a  wide  margin,  and  was  rated  last  by  55  percent  of  project 
managers.  In  the  comment  section  or  in  the  margins,  many  program  managers  wrote  that  the 
objectives  were  to  meet  the  cost  schedule  and  performance  goals  and  not  to  exceed  them. 

One  important  note  on  the  responses  was  that  over  20  percent  of  the  respondents  took  the 
time  either  to  write  comments  that  exceeding  the  objectives  had  little  or  no  impact  on  their 
performance  ratings  or  had  marked  all  four  objectives  as  the  fourth  and  lowest  priority.  These 
observations  were  supported  in  interviews  with  program  managers  and  with  a  separate  group  of 
junior  acquisition  Officers.  Many  officers  believe  that  the  performance  of  their  projects  does  not 
particularly  affect  their  performance  ratings. 
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Figure  10-12:  Perceived  Effect  of  Various  Objectives  on  Program  Managers’  Performance 
Ratings  and  Potential  for  Promotion  (Program  Office  Survey;  Number  of  Projects  =  142) 


Many  senior  acquisition  officers  commented  on  the  surveys  and  in  follow-up  interviews 
that  it  is  much  more  important  to  meet  the  expected  schedule,  no  matter  how  long,  than  to  try  to 
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achieve  a  shorter  schedule  and  risk  not  fulfilling  it.  Success  appears  to  be  defined  as  meeting  the 
planned  schedule  and  not  shortening  it.  Meeting  the  schedule  was  stated  to  be  more  important 
even  if  the  project  could  be  delivered  earlier  than  originally  planned.  The  perceived  success  of 
the  program  and  the  ramifications  for  the  personnel  involved  focused  on  how  the  project  met 
expectations,  not  on  the  time  required  to  develop  and  field  it.  Managing  expectations  for  the 
project  was  seen  as  a  significant  part  of  ensuring  the  project’s  perceived  success. 

An  example  of  managing  expectations  was  observed  in  a  case  study  of  a  successful  recent 
effort  to  develop  a  weapon  system.  The  project  received  very  high  Air  Force  priority  and  its 
funding  was  not  limited.  Several  earlier  tests  had  already  been  conducted  on  prototypes,  and  the 
project  relied  entirely  on  proven  technologies.  Even  in  this  case,  the  Program  Office  strongly 
resisted  significant  efforts  by  several  senior  leaders  to  shorten  the  project’s  initial  schedule 
during  the  planning  phase.  This  occurred  despite  the  fact  that  the  Program  had  strong  evidence 
that  a  significantly  shorter  schedule  was  possible,  and  a  contractor  that  pushed  for  and  initially 
bid  a  significantly  shorter  schedule.  The  program  manager  stated  that  resisting  the  shorter 
schedule  ensured  that  the  project  would  meet  the  schedule  thus  be  seen  as  a  success.  The 
program  manager  bluntly  stated  that  it  was  more  important  for  this  project  to  be  considered  a 
success  even  if  the  total  time  was  significantly  longer  than  to  shorten  the  planned  schedule  and 
risk  of  not  meeting  it.  The  extra  time  caused  a  two-year  delay  in  integrating  the  weapon  into  the 
flight  mission  software  of  several  fighter  aircraft,  as  the  schedule  moved  to  a  later  bi-annual 
update.  This  significantly  delayed  initial  operating  capability  on  those  aircraft,  and  the  system 
was  not  ready  during  a  threatened  action  against  Iraq  in  the  spring  of  1998.** 

During  a  discussion  group  with  approximately  20  junior  acquisition  officers  (captains 
and  lieutenants),  it  was  evident  that  the  group  felt  the  performance  of  the  project  they  worked  on 
had  little  impact  on  their  performance  ratings  or  potential  for  promotion."  Captains  and 
lieutenants  are  typically  assigned  as  project  managers  on  smaller  development  projects.  Several 
stated  outright,  and  most  agreed,  that  how  they  fulfilled  their  additional  duties,  such  as 


98  “US  Bombs  Not  Much  ‘Smarter’”  Boston  Globe.  Page  1.  February  20, 1998. 

"  The  meeting  was  conducted  with  20  officers  at  Hanscom  Air  Force  Base  in  September  1996.  It  was  intended  to 
address  the  concerns  of  the  junior  officers  in  the  acquisition  related  career  field. 
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organizing  a  unit  party,  had  a  larger  effect  on  their  performance  rating  then  the  performance  of 
their  program. 

One  potential  reason  for  this  apparent  lack  of  accountability  is  the  short  period  that  each 
project  manager  is  in  charge  of  a  project.  Some  65  percent  of  managers  surveyed  have  been  in 
charge  of  their  projects  for  less  than  18  months,  and  47  percent  have  been  in  charge  less  than  12 
months.  The  average  tenure  for  managers  of  the  144  projects  surveyed  is  20  months.  And  this 
figure  includes  a  number  of  civilian  program  managers,  who  typically  stay  significantly  longer 
than  their  military  counterparts.  The  average  tenure  is  short  compared  with  the  average  of  75 
months  from  start  to  first  production  item  for  the  same  projects.  The  result  is  a  large  number  of 
managers  per  project:  80  percent  of  projects  surveyed  have  changed  managers  at  least  once, 
while  57  percent  have  had  three  or  more  project  managers  so  far. 

A  large  number  of  project  managers  makes  for  difficulty  in  assigning  decision-making 
responsibility.  Even  if  responsibility  for  a  decision  can  be  assigned  to  a  particular  manager,  her 
or  she  has  often  moves  on  before  the  results  ensue.  In  fact,  it  is  standard  policy  to  ensure  that 
younger  officers,  who  often  serve  as  managers  on  small  projects,  change  Program  Offices  every 
18  months  to  ensure  that  they  experience  different  acquisition  positions  during  a  typical  three- 
year  assignment. 

The  survey  data  show  not  only  that  program  managers  change  positions  but  that  the  entire 
staff  rotates  out  as  various  projects  proceed,  undermining  continuity.  Figure  10-15  shows  the 
percentage  of  original  project  staff  remaining  on  a  project.  The  Packard  Commission  report 
raised  similar  concerns  10  years  ago. 


254  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


to 

<D 

U) 

(0 

C 

CO 


E 

03 

i— 

U) 

o 

CL 

*4— 

O 


<D 

n 

E 

u 


40 

35 

30 

25 

20 

15 

10 

5 

0 


O  1  I  I  »  I  i  •  •  *  I  *  '  I  *  ,  ^ 

'  •  •  '  '  1  '  ' 

r-t-N(0C0^'flfl(0®NL(DCD(CNC0'9-Offl(M(D 

0)OOrNN(0(0 


Months  in  Current  Project  Manager  Position 


Figure  10-13:  Project  Manager  Tenure  in  Current  Position.  (Program  Office  Survey; 
Number  of  Projects  =  144) 
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Since  Project  Initiation 

Figure  10-14:  Number  of  Government  Program  Managers  per  Project  (Program  Office 
Survey;  Number  of  Projects  =  138) 
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Figure  10-15:  Percent  of  Original  Staff  on  a  Project  (Program  Office  Survey;  Number  of 
Projects  =  138) 

D.  Contractors’  Incentives  to  Reduce  Schedule 

The  Program  Office  and  contractor  surveys  asked  questions  to  determine  contractors’ 
incentives  to  meet  or  shorten  a  project’s  schedule.  The  Program  Office  survey  asked  project 
managers  to  indicate  the  schedule-related  incentives  they  provide  to  contractors  through  their 
contracts.  The  contractor  survey  asked  contractors  to  indicate  the  incentives  from  Program 
Offices,  and  their  overall  incentives  to  meet  or  exceed  project  objectives  based  on  their 
companies’  bottom  line.  The  results  indicate  that  contractors  have  few  incentives  to  shorten 
project  schedules. 


D.l  Program  Office  Incentives  for  Contractors 

The  Program  Office  survey  asked  questions  about  schedule-related  incentives  the  Program 
Office  provided  to  contractors  for  on-time  or  early  completion  of  a  project  or  major  milestone. 
The  vast  majority  of  project  managers  report  no  financial  incentives  for  contractors  to  either 
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meet  or  reduce  the  project  schedule.  Two-thirds  of  all  projects  include  no  financial  incentives 
for  on-time  completion.  Some  75  percent  of  project  managers  report  that  their  contracts  provide 
no  financial  incentive  for  early  completion,  while  85  percent  report  that  the  incentive  for  early 
completion  was  less  than  2  percent  of  the  contract  value.  The  average  incentive  for  on-time 
completion  across  all  projects  is  less  than  3  percent  of  the  contract  value.  The  average  incentive 
available  for  early  completion  of  a  project  or  major  milestone  is  less  than  1.5  percent  of  the 
contract  value. 

Despite  the  low  schedule-related  incentives  reported,  these  numbers  may  significantly 
overestimate  the  schedule-based  incentives.  Written  comments  and  follow-up  interviews 
revealed  that  many  respondents  included  the  entire  potential  award  fee  in  their  responses,  and 
not  the  percentage  of  the  award  fee  associated  solely  with  schedule  performance.  The  award  fee 
is  typically  based  on  a  number  of  factors,  including  schedule.  A  contractor  may  receive  all  or 
nearly  all  of  the  incentive  because  of  considerations  other  than  schedule.  This  would  imply  that 
the  amount  of  incentive  specifically  available  for  on-schedule  performance  or  shortened 
schedule  may  be  even  less  than  these  small  averages  indicate.  The  average  percentage  of 
available  award  fees  given  to  contractors  is  near  93  percent.'00 


100  Based  on  conversations  with  Mr.  Blaise  Durante.  Deputy  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
Management  Policy  and  Program  Integration.  SAF/AQX. 
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Available  Incentive  for  On-Time  Completion 


Figure  10-16:  Percentage  of  Contract  Value  Available  for  Incentive  for  On-Time 
Completion  of  the  Project  or  a  Major  Milestone.  Program  Office  Survey.  Number  of 
Projects  =  108. 
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Figure  10-17:  Percentage  of  Contract  Value  Available  for  Early  Completion  of  a  Project  or 
Major  Milestone  (Program  Office  Survey;  Number  of  Projects  =108) 
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Comments  from  several  industry  representatives  involved  in  the  Lean  Aerospace 
Initiative  indicate  that  award  fees  may  play  a  significantly  larger  role  than  indicated  by  financial 
aspects.  The  percentage  award  fee  is  in  essence  a  report  card  from  the  Program  Office  on  the 
company’s  response  to  the  Program  Office’s  desires  and  concerns.  Senior  managers  at  the 
company  use  the  percentage  of  the  award  fee  to  indicate  the  project’s  status. 

Another  issue  is  that  contract  incentives  for  on-time  completion  are  based  on  the  schedule 
as  it  exists  at  the  time  of  evaluation  or  completion,  not  the  original  contracted  schedule.  The 
contracted  schedule  can  and  often  does  change  by  mutual  agreement.  This  allows  contractors  to 
be  “on  schedule”  despite  to  the  fact  that  the  schedule  has  lengthen  significantly. 

The  observation  that  there  is  not  a  significant  incentive  schedule  fulfillment  and  reduction 
is  consistent  with  the  low  reported  priority  placed  those  ends  by  the  Program  Offices. 


D.2  Program  Office  Incentives  as  Viewed  by  Contractors 

The  Contractor  survey  asked  respondents  to  rate  the  extent  that  a  Program  Office  provides 
specific  financial  or  other  incentives  to  reduce  total  project  cost,  cut  unit  cost,  shorten  schedule, 
enhance  technical  performance,  and  increase  reliability  and  ease  of  maintenance.  A  large 
majority  of  respondents  reported  that  the  Program  Offices  provide  no  incentive  for  exceeding  any 
program  objectives.  Most  incentives  appear  to  be  based  on  meeting  the  stated  project  goals  and 
not  exceeding  them.  Incentives  to  reduce  total  program  cost  were  rated  higher  than  reducing  unit 
cost,  exceeding  performance  requirements,  or  shortening  schedule — all  of  which  were 
statistically  equivalent.  Incentives  to  exceed  operational  and  maintenance  requirements  were 
rated  below  the  others.  When  viewed  across  all  projects,  none  of  the  incentives  were  rated  as 
greater  than  small,  with  a  large  majority  of  projects  carrying  no  customer-provided  incentives  to 
exceed  project  goals. 
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Figure  10-18:  Program  Office-  Incentives  for  Exceeding  Project  Goals  as  Viewed  by 
Contactors  (Number  of  Projects  =102) 
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Figure  10-19:  Average  Contractor  Incentives  Provided  by  Program  Office  for  Exceeding 
Project  Goals  (Contractor  Survey;  Number  of  Projects  =  102) 
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Some  73  percent  of  contractor  project  managers  indicated  that  the  Program  Office  did  not 
provide  any  incentives,  or  in  some  cases  provided  negative  incentives,  to  shorten  schedule.  Only 
18  percent  indicated  that  the  customer  provided  more  than  a  slight  incentive  to  shorten  project 
schedules.  These  findings  are  consistent  with  the  responses  to  the  Program  Office  survey 
indicating  that  a  vast  majority  of  Program  Offices  do  not  provide  any  schedule-related  incentives. 
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Figure  10-20:  Contractor  Preceived  Incentives  for  Reducing  Schedule  -  Distribution  of 
Responses  from  Contractors  on  the  Incentives  Provided  by  the  Program  Office- 
(Contractor  Survey;  Number  of  Projects  =  102) 


The  type  of  contract  (cost-plus  or  fixed-price),  and  the  selection  method  (sole-source  or 
competitive),  did  not  significantly  affect  contractors’  incentives.  Most  working  under  both  fixed- 
price  and  on  cost-plus  contracts  indicated  a  very  small  incentive  to  shorten  project  schedules. 
Similarly,  companies  working  under  competitive-selection  and  sole-source  contracts  report  few 
incentives  to  shorten  project  schedules.  There  do  not  appear  to  be  any  significant  customer- 
provided  incentives  to  shorten  project  schedules  even  when  there  is  direct  competition  to  win  a 
contract.  Respondents  represented  roughly  equal  numbers  of  fixed-price  (41)  and  cost-plus  (46) 
contracts,  and  equal  numbers  of  competitive  (53)  and  sole-source  (46)  contracts. 
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Figure  10-21:  Perceived  Program  Office  Incentives  to  Complete  Project  Ahead  of  Schedule 
for  Fixed-Price  and  Cost-Plus  Contracts  (Contractor  Survey;  Number  of  Projects  =  91) 
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Figure  10-22:  Level  of  Customer-Provided  Incentives  to  Complete  Projects  Ahead  of 
Schedule  for  Competitive  and  Sole-Source  Contracts  (Contractor  Survey;  Number  of 
Projects  =  99) 
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D.3  Contractor’s  Bottom-Line  Incentives 

To  measure  a  company’s  overall  or  total  incentive  associated  with  a  project,  the 
Contractor  survey  asked  project  managers  to  indicate  the  bottom-line  impact  or  net  financial 
impact  of  exceeding  various  project  goals  on  their  company.  Such  impacts  may  include  follow- 
on  business  and  factors  such  as  increased  company  stature.  These  incentives  were  reported  to  be 
significantly  higher  than  those  provided  by  the  Program  Office.  The  objectives  with  the  highest 
incentives  were  reducing  total  program  cost  and  reducing  unit  costs.  Shortening  schedule  was 
rated  significantly  lower  than  cost  but  above  exceeding  performance  and  reliability  requirements. 
The  goals  of  exceeding  performance  and  maintenance  requirements  were  statistically  equivalent 
and  rated  significantly  lower  than  other  objectives.  A  large  percentage  of  respondents  indicated 
that  they  had  no  net  or  overall  incentive  to  exceed  performance  or  maintenance  requirements. 
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Figure  10-23:  Contractors’  Total  “Bottom-Line”  Incentives  for  Exceeding  Various  Project 
Goals  (Contractors  Survey;  Number  of  Projects  =  101) 
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Figure  10-24:  Average  “Bottom-Line”  Incentives  to  Exceed  Project  Objectives  (Contractor 
Survey;  Number  of  Projects  =  101) 

When  looking  specifically  at  schedule  related  incentives,  59  percent  of  contractors  report 
little  or  no  incentive  to  reduce  schedule.  41  percent  indicate  that  they  had  that  they  do  feel  some 
form  of  overall  incentive  to  reduce  schedule.  During  follow-up  interviews,  several  project 
managers  indicated  that  reducing  schedule  is  seen  as  integral  to  reducing  cost.  This  is  consistent 
with  the  results  of  analysis  of  the  reported  incentive  by  the  contract  type,  either  cost-plus  or 
fixed-price. 
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Figure  10-25:  Distribution  of  Responses  for  Bottom-Line  Impact  of  Completing  Ahead  of 
Schedule  (Contractor  Survey;  Number  of  Projects  =  101) 


The  type  of  contract  between  Program  Office  and  contractor  played  a  significant  role  in 
determining  contractors’  total  incentive  to  exceed  project  objectives.  Companies  with  fixed- 
price  contracts  account  for  a  disproportionate  share  of  those  reporting  incentives  to  reduce  total 
program  cost,  reduce  unit  cost,  and  shorten  schedule.  Companies  with  fixed-price  contracts 
reported  significantly  higher  net  incentives  to  exceed  cost  goals  than  those  with  cost-plus 
contracts.  Companies  with  cost-plus  contracts  report  only  a  slight  net  incentive  to  reduce  costs  or 
shorten  schedule.  Most  companies  with  cost-plus  contracts  report  Program  Office-  incentives 
and  bottom-line  incentives  as  equal,  indicating  that  these  companies  have  little  internal  incentive 
to  exceed  project  goals.  Companies  working  on  projects  with  fixed-price  contracts  reported  a 
slightly  higher  incentive  to  reduce  schedule  than  those  with  cost-plus  contracts.  Both  companies 
with  cost-plus  contracts  and  those  with  fixed-price  contracts  reported  few  incentives  to  exceed 
performance  and  reliability  goals. 
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Figure  10-26:  Contractors’  Reported  Bottom-Line  Incentives  to  Exceed  Project  Cost  and 
Schedule  Goals  by  Project  Type  (Number  of  Projects  =  101) 
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Figure  10-27:  Contractors’  Reported  Bottom-Line  Incentives  to  Exceed  Project  Performance 
and  Requirement  Goals  by  Contract  Type  (Number  of  Projects  =  101) 
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Figure  10-28:  Average  of  Contractor  Reported  Net  or  Bottom-Line  Incentives  by  Fixed- 
Price  and  Cost-Plus  Contract  Type  (N  =  101) 
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Summary  of  Schedule-Related  Incentives 

Pentagon  respondents  report  that  users  have  significant  incentives  to  exceed  program 
cost,  schedule,  performance,  and  reliability  goals.  Users'  incentive  to  reduce  project  schedule  is 
rated  slightly  higher  than  users’  incentives  to  reduce  cost  or  increase  performance.  Overall  80 
percent  of  projects  are  reportedly  desired  as  soon  as  possible,  and  73  percent  are  reportedly 
needed  to  meet  current  operational  requirements. 

Analysis  of  Pentagon  responses  indicates  few  incentives  at  the  Pentagon  level  for  reducing 
schedule  in  the  majority  of  development  projects.  The  principle  Pentagon  actors  on  individual 
development  efforts — the  program  element  monitors  and  the  Air  Staff  action  officers — do  not 
see  shortening  cycle  time  as  either  an  important  project  or  personal  objective.  They  rate 
shortening  schedules  as  the  fourth  of  four  project  objectives,  and  indicate  that  shortening 
schedules  has  little  impact  on  their  performance  rating  or  potential  for  promotion.  Higher-level 
Pentagon  actors,  such  as  mission  area  directors,  oversee  many  development  efforts,  and  the 
outcome  of  a  single  project  has  less  of  an  impact  on  their  careers. 

As  shown,  few  incentives  within  a  Program  Office  aim  to  shorten  a  project’s  schedule. 
Incentives  for  project  managers  are  to  meet  the  project  schedule  and  not  to  shorten  it.  The 
Program  Offices  report  shorter  schedules  most  often  as  the  fourth  of  four  objectives,  and  do  not 
rate  short  schedules  as  particularly  important  on  its  own.  Shortening  the  project  schedule  is  not 
seen  as  having  a  significant  impact  on  a  program  manager’s  performance  rating  or  career,  and 
there  is  apparently  little  personal  accountability  for  project  performance.  Given  the  multiplicity 
of  project  managers  per  project,  it  is  difficult  to  assign  either  credit  or  blame  to  a  project  manager 
based  on  a  project’s  performance.  Across  most  projects  there  appears  to  be  little  incentive  to 
reduce  project  schedules  within  the  Program  Offices. 

The  Program  Offices  also  provide  few  incentives  for  contractors  to  exceed  project 
objectives.  Meeting  the  objectives--not  beating  them-appears  to  be  the  overall  goal  in  a 
majority  of  projects.  Program  Offices  report  providing  practically  no  contract-based  incentives 
for  either  on-time  or  early  completion  of  a  major  milestone  or  a  project.  Incentives  perceived  by 
contractors  focus  on  cost  and  depend  largely  on  the  type  of  contract.  Projects  with  fixed-price 
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contracts  account  for  a  large  portion  of  reported  net  incentives  to  reduce  cost  by  reducing 
schedule.  Projects  on  cost-plus  contracts  report  little  Program  Office  or  overall  incentive  to 
exceed  project  goals. 

The  lack  of  personal  and  organizational  incentives  for  defense  contractors,  Program 
Offices,  and  the  Pentagon  indicates  that  the  goal  is  to  meet  project  objectives,  not  to  exceed 
them.  There  is  little  perceived  incentive  to  exceed  a  project’s  objectives  at  any  level.  Reducing 
schedules  is  most  often  the  last  of  project  objectives.  Additional  efforts  and  resources,  if 
available,  would  likely  be  placed  on  the  higher-rated  objectives  of  reducing  cost  or  improving 
performance,  not  on  reducing  project  schedules.  The  effects  of  the  low  priority  accorded  to 
shortening  development  schedules,  and  the  lack  of  incentives  for  doing  so,  can  be  seen  in  the 
schedules’  execution. 

The  original  schedule  is  based  not  on  a  project’s  development  requirements  but  on 
expected  funding  limitations.  In  the  contracting  phase,  the  original  DoD  project  schedule  is  the 
central  factor  in  determining  the  contract  schedule;  contractors  have  no  incentive  to  bid  anything 
other  than  what  the  government  expects.  Nor  is  there  any  mechanism  to  rectify  the  influence  of 
the  original  project  schedule  on  development  To  reduce  development  times,  DoD  must  improve, 
redesign,  or  dramatically  alter  the  process  used  to  develop  project  schedules. 

The  impact  of  the  project  scheduling  processes  and  the  lack  of  schedule-related  incentives 
can  be  seen  in  the  results  of  the  projects  surveyed.  The  impacts  include  few  accelerated  projects, 
many  delayed  projects,  project  instability,  and  higher  costs.  These  impacts  are  addressed  in  the 
next  chapter. 


Chapter  11 


Executing  Project  Schedules  during  the 
Development  Phase _ 


Although  the  contracted  schedule  plays  a  significant  role  in  product  development  times,  those 
times  can  grow  or  shrink  depending  on  technical  problems,  funding  constraints,  and  changing 
Air  Force  objectives.  The  execution  of  a  project’s  development  schedule  also  depends  on  the 
appropriateness  of  the  schedule,  the  contractor’s  execution,  and  responses  of  the  Program  Office 
and  the  contractor  to  unforeseen  events.  The  responses  to  unforeseen  events,  in  turn,  depend  on 
the  schedule-related  incentives  experienced  by  various  organizations. 

This  chapter  looks  at  the  execution  of  project  schedule  during  development.  First,  it 
investigates  differences  among  the  planned  time,  the  achieved  schedule,  and  the  estimated  time 
required  to  develop  the  projects  based  on  their  technical  requirements.  It  then  looks  at  the  causes 
of  deviations  from  project  plans,  and  the  tradeoffs  made  in  response  to  changes  in  the  plan.  It 
identifies  how  often  and  to  what  extent  these  changes  occur.  This  chapter  also  identifies  the 
primary  barriers  to  shortening  project  schedules. 
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A.  Achieved  Project  Schedules  in  the  Development  Phase 

The  extent  to  which  projects  achieve  their  planned  schedule  is  one  measure  of  the 
effectiveness  of  the  schedule  development  process.  Another  measure  is  planned  time  planned 
versus  the  time  posited  to  be  required  to  perform  the  development-related  activities.  Technical 
barriers  would  indicate  that  the  schedule  development  process  underestimated  the  necessary 
time. .  Large  differences  between  the  project  schedule  and  the  estimated  time  needed  to  fulfill 
the  requirements  would  indicate  significant  slack  in  project  schedules  and  longer-than-necessary 
development  times. 


A.l.  Achieved  versus  Planned  Project  Schedules 

To  compare  achieved  times  with  planned  project  schedules,  the  three  surveys  asked 
respondents  how  much  their  project  had  changed  from  its  initial  schedule.  The  amount  of 
schedule  change  was  based  on  delivery  of  the  first  production  item  or  other  appropriate  milestone 
versus  the  initial  project  plan.  The  surveys  also  asked  respondents  to  provide  the  dates  of  key 
milestones  for  both  the  initial  and  the  current  schedule. 

The  responses  indicated  a  large  number  of  project  slips.  The  average  project  slip 
measured  across  all  projects  surveyed  was  12.1  months.  This  might  indicate  that  planned 
schedules  may  be  too  short  to  allow  for  required  development  activities.  The  average  schedule 
slip,  however,  is  misleading,  as  a  significant  fraction  of  projects  do  not  slip.  Among  those  that 
do  slip,  most  schedule  changes  are  reportedly  due  to  external  factors  and  not  to  technical 
problems.  The  surveys  indicate  that  106  of  271  projects-or  39  percent-  are  within  3  months  of 
their  initial  schedule.  Some  77  projects,  or  28  percent,  are  reportedly  between  3  months  and  1 
year  of  their  initial  schedule.  Only  57  of  the  271  projects,  or  27  percent,  are  reportedly  more  than 
1  year  behind  their  initial  schedule. 

Thesurvey  results  also  show  that  despite  the  large  number  of  development  projects  that 
are  exactly  on  schedule,  few  are  ever  completed  early.  Of  the  271  projects,  only  14— or  5.2 
percent— were  reportedly  ahead  of  their  initial  schedule  by  3  months  or  more. 
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Figure  11-1:  Project  Schedule  Performance  Based  on  Initial  Project  Plan  (Pentagon, 
Program  Office,  and  Contractor  Surveys;  Number  of  Projects  =  271) 


Differences  in  project  duration  do  not  account  for  the  amount  of  project  slip.  The  average 
annual  slip  for  the  225  projects  whose  managers  provided  the  necessary  information  is  2.71 
months.  This  indicates  that  in  each  year  of  development,  the  average  project  makes  only  slightly 
more  than  9  months  of  progress  toward  completion.  A  few  projects- 13,  or  5.8  percent— 
reportedly  had  annual  schedule  slips  of  12  or  more  months.  This  indicates  that  those  projects  are 
getting  further  from  first  production  delivery  every  year.  A  project’s  large  average  annual  slip 
may  result  from  significant  slip  early  in  its  development.  Calculated  annual  project  slips  are 
shown  in  Figure  1 1-2. 
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Annual  Program  Slip  (Months  Per  Year) 

Figure  11-2:  Annualized  Project  Schedule  Performance  Based  on  Initial  Project  Plan 
(Pentagon,  Program  Office,  and  Contractor  Surveys;  Number  of  Projects=225) 


As  Figures  11-1  and  1 1-2  indicate,  a  large  number  of  projects  are  exactly  on  schedule. 
There  are  many  possible  reasons  for  this  record.  Despite  the  clear  wording  of  the  questionnaires, 
respondents  may  have  compared  their  project’s  performance  to  the  current  schedule— which  had 
already  been  modified  from  the  initial  schedule.  Another  reason  may  be  that  project  managers 
often  do  not  admit  to  slips  until  the  final  stages,  when  they  realize  that  they  cannot  recover  from 
earlier  shortcomings.  However,  respondents’  answers  to  other  questions  provided  support  for  the 
results  as  presented.  The  slip  for  projects  calculated  from  the  initial  and  current  schedules  are 
similar  to  respondents’  estimates  of  the  amount  of  slip  from  the  initial  plan. 


Number  of  Projects 


11.  Executed.  Project  Schedules  During  the  Development  Phase 


273 


Slip  of  First  Production  Item  (Months) 


Figure  11-3:  Slip  of  First  Production  Item  as  Calculated  from  Calendar  Dates  Provided  for 
Initial  and  Current  Project  Plans  (Program  Office  Survey;  Number  of  Projects  =  52) 


Similar  results  appear  in  databases  from  other  sources.  The  RAND  Corporation  collected 
information  on  all  major  defense  acquisition  projects  since  1960  as  reported  in  selected 
acquisition  reports  to  Congress.  Of  those,  some  100  projects  included  the  information  needed  to 
calculate  the  amount  of  slip  from  project  initiation  to  first  production  item  or  operational 
delivery.  The  slip  per  project—and  the  distribution  of  slip  across  many  projects— are  equivalent  to 
those  found  in  the  three  surveys. 
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Figure  11-4:  Total  Slip  (from  Project  Initiation  to  First  Operational  Delivery)  for  Major 
Defense  Acquisition  Projects,  Based  on  the  Rand  Selected  Acquisition  Report  Database1*1 

A.2.  Causes  of  Project  Schedule  Slip 

The  simple  number  of  projects  that  slip,  and  the  amount  of  schedule  slip,  do  not  provide  a 
complete  picture  of  the  ability  of  a  project  to  meet  its  initial  schedule.  Completing  the  picture 
requires  examining  the  causes  of  schedule  slips.  To  determine  what  is  causing  schedule  slips,  the 
three  surveys  asked  respondents  to  estimate  the  percentage  of  any  slip  stemming  from  funding 
instability,  requirement  changes,  technical  problems,  or  other  factors.  The  impact  of  these 
factors  can  be  seen  in  the  magnitude  of  their  effect  across  projects,  and  in  the  number  of  projects 
they  affect. 

The  average  amount  of  slip  for  projects  at  least  1  year  into  development  was  12.1  months. 
Respondents  attributed  5.2  of  those  months  to  funding  instability,  3.1  months  to  technical 
problems,  2.6  months  to  requirement  changes,  and  1.2  months  to  “other  causes.”  The  average 


101  Jarvaise,  Drezner,  and  Norton.  “The  Defense  System  Cost  Performance  Database:  Cost  Growth  Analysis  Using 
Selected  Acquisition  Reports.”  MR-625-OSD  Santa  Monica  CA:  The  RAND  Corporation.  1996. 
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schedule  slip  attributed  to  funding  instability  across  all  programs  indicates  that  it  is  the  primary 
cause. 


Program  Slip  Technical  Problems  Other  Causes 

Funding  Instability  Requirement  Changes 


Figure  11-5:  Average  Slip  for  Projects  at  Least  1  Year  into  the  Development  Phase,  and  the 
Portions  Attributed  to  Different  Causes  (Pentagon,  Program  Office,  and  Contractor 
Surveys;  Number  of  Projects  =  271) 

Analyzing  the  data  on  an  annualized  basis  yields  similar  results.  Across  all  projects  in 
development  for  at  least  one  year,  the  average  project  slip  was  2.1  months  per  year.  Of  those 
months,  1.0  per  year  was  attributed  by  respondents  to  funding  instability,  0.5  per  year  to 
technical  problems,  0.4  per  year  to  requirement  changes,  and  0.2  month  per  year  to  other  causes. 
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Figure  11-6:  Average  Annual  Slip  for  Projects  at  Least  1  Year  into  the  Development  Phase, 
and  the  Portions  Attributed  to  Different  Causes  (Pentagon,  Program  Office,  and 
Contractor  Surveys;  Number  of  Projects  =  175) 


Respondents  reported  that  technical  problems  account  for  only  25  percent  of  the  observed 
slip.  Factors  not  related  to  technical  requirements,  including  funding  stability,  requirement 
changes,  and  “other  causes,”  account  for  75  percent  of  the  total  amount  of  slip.  These  results  are 
also  observed  when  slip  is  viewed  on  an  annualized  basis.  Of  course,  unlike  technical  factors, 
funding  instability,  requirement  changes,  and  “other  causes”  are  typically  outside  the  control  of 
Program  Offices  and  contractors. 

In  number  of  projects,  84  of  271  reported  no  schedule  slip,  and  120  reported  less  than  3 
months’  slip.  Some  107  attributed  some  portion  of  their  schedule  delay  to  technical  problems,  97 
attributed  some  delay  to  funding  instability,  89  attributed  some  delay  to  requirement  changes, 
and  65  projects  attributed  some  delay  to  “other  factors.”  Of  projects  attributing  slip  primarily  (at 
least  50  percent)  to  a  single  cause,  62  pointed  to  funding  instability,  54  reported  technical 
problems,  35  projects  pointed  to  requirements  changes,  and  33  projects  reported  other  factors.102 
Thirty-five  projects  reported  funding  instability  as  solely  responsible  for  the  schedule  slip,  and 


102  Those  projects  not  attributing  50%  to  any  factor  are  not  included  and  those  attributing  50%  for  two  factors  are 
included  twice. 
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20  projects  reported  technical  problems  as  solely  responsible.  For  130  of  187  projects  reporting 
any  schedule  slip,  external  factors  such  as  funding  instability,  requirement  changes,  and  other 
factors  were  the  primary  cause. 

In  terms  of  technical  problems,  the  responses  indicate  that  105,  or  39  percent,  of  all  projects 
attributed  some  portion  of  schedule  slip  to  such  problems.  Some  164,  or  61  percent,  of  all 
projects  reported  no  schedule  delay  stemming  from  technical  problems.  Of  the  107  projects 
reporting  some  delays  attributable  to  technical  problems,  54  projects  indicated  that  they  were  not 
the  primary  cause  (accounting  for  less  than  50  percent).  Of  the  projects  that  attributed  schedule 
delays  primarily  to  technical  problems,  only  14  had  delays  longer  than  1  year.  This  compares 
with  33  projects  with  over  1  year  of  delay  that  reported  funding  instability  as  the  primary  cause. 
In  all  40  of  the  54  projects  that  reported  slip  primarily  owing  to  technical  problems  report  12 
months  of  delay  or  less.  Twenty-four  of  those  projects  reported  delays  of  6  months  or  less. 
Overall,  198  of  271  projects— or  73  percent— attributed  25%  percent  or  less  of  schedule  slip  to 
technical  problems. 


Percent  of 
Project  Slip 
Attributed 

Funding 

Instability 

Technical 

Problems 

Requirement 

Changes 

Other 

174 

164 

182 

206 

E33E5SS3BBI 

97 

107 

89 

65 

25%  or  Greater 

76 

73 

59 

46 

50%  or  Greater 

62 

54 

35 

33 

75%  or  Greater 

36 

32 

17 

24 

100%  of  Slip 

35 

17 

8 

21 

Table  11-17:  Number  of  Projects  Attributing  Schedule  Slip  to  Funding  Instability, 
Technical  Problems,  Requirement  Changes,  and  Other  Causes  (Pentagon,  Program  Office, 
and  Contractor  Surveys;  Number  of  Projects  =  271)1M 


,ra  Note:  Those  projects  not  attributing  50%  to  any  factor  are  not  included  and  those  attributing  50%  for  two  factors 
are  included  twice. 
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Figure  11-7:  Number  of  Projects  Attributing  Percentage  of  Schedule  Slip  to  Various 
Causes  (Pentagon,  Program  Office,  and  Contractor  Surveys;  Number  of  Projects  =  271) 


According  to  the  survey  responses  shown  in  Figures  1 1-5  and  1 1-6,  the  primary  cause  of  project 
slip  is  funding  instability,  which  underlies  half  of  the  total  reported  slip.  Factors  external  to 
development  requirements  are  causing  75  percent  of  all  schedule  delays.  The  fact  that  only  1  in 
5  projects  reported  slip  primarily  (over  50  percent)  stemming  from  technical  problems  indicates 
that  many  schedules  are  not  set  too  short  based  on  development  requirements.  This  is  entirely 
consistent  with  earlier  findings  that  development-related  considerations  were  secondary  to 
budgetary  factors  in  setting  schedules  during  the  planning  phase. 

The  fact  that  3  of  5  current  development  projects  did  not  experience  any  schedule  slip 
owing  to  technical  problems  may  indicate  that  many  managers  may  have  been  able  to  cut  slack  in 
their  schedules  without  changing  project  requirements.  This  is  consistent  with  the  finding  in 
Chapter  8  that  77  percent  of  all  projects  are  funding  limited,  while  only  23  percent  are 
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constrained  by  technological  and  engineering  requirements.  It  may  be  possible  to  shorten 
development  time  so  that  technical  requirements  are  the  determining  factor,  without  increasing 
technical  risk  or  sacrificing  performance.  This  could  be  done  by  aligning  resources  with  actual 
development  requirements  and  removing  funding-based  schedule  limits. 

This  analysis  indicates  only  the  causes  of  schedule  slip  —  not  whether  schedules  are 
planned  longer  than  necessary.  As  shown  earlier,  technical  requirements  are  not  the  primary  * 
concern  in  establishing  initial  project  schedules,  nor  are  they  contractors’  primary  consideration 
in  proposing  schedules.  Development-related  requirements  are  not  the  cause  of  schedule  delays 
for  a  majority  of  projects.  This  would  be  expected  if  schedules  were  either  set  appropriately  or 
were  longer  than  necessary  for  the  planned  activities. 

B.  Required  Versus  Scheduled  Development  Time 

It  is  difficult  to  determine  the  minimum  time  required  to  develop  a  product  based  on  its 
technical  requirements  without  a  detailed  understanding  of  the  project  and  significant  experience 
on  similar  projects.  To  measure  the  time  required  to  develop  projects  based  on  technical 
requirements,  the  Program  Office  and  Pentagon  surveys  asked  respondents  how  long  they 
believed  it  would  take  to  deliver  the  first  production  item,  given  strong  incentives  and  proper 
resources.  While  not  a  definitive  measure  of  required  time,  the  answers  indicate  that  respondents 
believe  projects  could  be  completed  in  significantly  less  time  than  currently  planned. 

The  Program  Office  survey  asked  project  managers  how  long  it  would  take  to  field  the 
first  system  if  it  was  deemed  as  essential  in  a  war.  The  responses  were  compared  with  the  time 
remaining  in  the  schedule  until  delivery  of  the  first  production  item.  Of  the  37  projects  with  1 
year  or  more  remaining  in  development,  project  managers  estimated  the  time  required  to  field  the 
first  production  item  was  52  percent  of  the  current  schedule.  Thus,  the  minimum  time  required 
to  develop  a  project  is  significantly  shorter  than  most  schedules.  No  project  manager  reported 
that  their  project  schedule  could  not  be  shortened. 
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Figure  11-8:  Percent  of  Remaining  Development  Schedule  Required  by  Project  Managers 
to  Field  First  Production  Unit  in  a  Wartime  Scenario  (Program  Office  Survey;  Number  of 
Projects  =  37) 


Pentagon  survey  respondents  were  asked  to  provide  the  amount  of  time  required  from 
program  start  to  first  production  item,  assuming  no  funding  constraints  and  based  on  the  project’s 
record  to  date.  Twenty-six  of  35  respondents  thought  that  projects  could  be  completed  faster 
than  scheduled.  Only  6project  managers  thought  fielding  the  first  production  item  would  require 
the  scheduled  time.  Three  respondents  indicated  that  significantly  longer  time  was  required  than 
that  available,  indicating  that  a  schedule  slip  would  likely  occur.  These  responses  indicated  that, 
in  the  absence  of  funding  constraints,  on  average  73  percent  of  the  current  schedule  was  needed 
to  complete  current  set  of  projects. 
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Ratio  of  Total  Project  Schedule  Required 

Figure  11-9:  Ratio  of  Planned  Schedule  to  Required  Time  Estimated  by  Pentagon 
Respondents,  Assuming  No  Resource  Constraints  (Number  of  Projects  =  35) 


Program  Office  and  Pentagon  respondents  did  not  base  the  ability  to  accelerate  projects 
on  a  different  technical  approach.  Thus  these  responses  may  significantly  underestimate 
reductions  in  development  times  that  are  possible. 

While  these  results  do  not  provide  firm  evidence  that  projects  could  be  completed  faster 
than  currently  planned,  they  do  indicate  that  the  people  most  closely  involved  believe  that  could 
occur  without  loss  of  technical  capability.  This  indicates  that  Program  Office  and  Pentagon 
respondents  believe  the  time  allowed  in  current  schedules  is  significantly  longer  than  required  to 
accomplish  the  specific  development  tasks.  The  data  suggest  that  reducing  funding-based 
constraints  and  focusing  on  project  schedules  could  cut  development  times  by  25  to  50  percent — 
without  changing  the  character  of  the  projects. 
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Experiences  the  commercial  sector  support  the  notion  that  a  focus  on  cycle  time  can  yield 
dramatic  cuts  in  development  times  and  costs,  boost  product  quality,  and  help  companies  meet 
customers’  needs.  Many  commercial  firms  have  cut  development  times  from  50  to  70  percent.'04 


C.  Causes  of  Changes  in  Project  Plans 

To  determine  the  major  causes  of  changes  in  plans,  the  Program  Office  and  Contractor 
surveys  asked  respondents  to  rate  potential  sources  of  project  instability.  The  list  was  compiled 
from  available  literature  and  interviews  with  program  managers.  The  surveys  asked  respondents 
to  rate  the  impact  of  each  factor  on  their  project,  from  “not  a  factor”  to  “the  primary  factor.”  The 
results  show  that  the  most  significant  causes  of  program  instability  are  changes  in  annual  budget 
allocations.  Rated  significantly  lower  were  unanticipated  technical  challenges  and  changes  in 
users’  requirements.  Still  lower  were  long  acquisition  cycles,  staffing  changes  among 
contractors  and  Program  Offices,  changing  service  priorities,  and  poor  contractor  performance. 
Rated  lowest  were  technical  problems  in  associated  projects. 


104  Dr.  Chris  Meyers.  “Lessons  From  Industry”  Lean  Aerospace  Initiative  Plenary  Session.  Hartford  CT.  Sept  1997. 
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Figure  11-11:  Average  Responses  on  the  Causes  of  Program  Instability  (Program  Office 
and  Contractor  Surveys;  Number  of  Projects  =  245) 


Of  the  factors  affecting  project  instability,  budget  allocations  were  rated  a  larger  factor 
than  technical  challenges  in  124  projects,  while  technical  challenges  were  rated  as  more 
important  in  86  projects.  Thirty-five  project  managers  rated  them  the  same.  Changes  in  budget 
allocation  had  more  effect  than  changes  in  user  requirements  in  1 15  projects,  while  changes  in 
user  requirements  rated  higher  in  93  projects.  Changes  in  requirements  and  technical  challenges 
were  rated  roughly  equal,  with  97  project  managers  indicating  technical  problems  and  90 
managers  indicating  user  requirement  changes  as  exerting  a  larger  effect.  Sixty-one  project 
managers  rated  them  equal.  This  indicates  that  budget  instability  is  the  most  prominent  factor  in 
project  instability,  but  technical  problems  and  changes  in  user  requirements  also  play  a 
significant  role. 
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Figure  11-12:  Effect  of  Program  Instability  on  Ability  to  Meet  Projects’  Overall  Objectives 
(Pentagon,  Program  Office,  and  Contractor  Surveys;  Number  of  Projects  =  309) 


Pentagon  respondents  indicated  less  effect  from  funding  and  technical  instability  than 
Program  Office  and  contractor  respondents.  Forty  percent  of  Pentagon  respondents  indicated 
that  program  instability  had  little  effect,  while  25  percent  of  contractors  and  16  percent  of 
government  project  managers  indicated  little  effect.  Thirty-five  percent  of  both  government  and 
contractor  project  managers  indicated  that  program  instability  had  a  significant  or  major  impact 
on  their  project's  ability  to  meet  its  goals,  while  24  percent  of  Pentagon  respondents  indicted  a 
similar  level  of  impact. 
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Figure  11-13:  Effects  of  Program  Instability  Reported  by  Government  Program  Managers, 
Contractor  Program  Managers,  and  Pentagon-Level  Respondents 


The  contractor  and  Pentagon  surveys  asked  to  what  extent  managers  could  head  off 
program  instability  before  it  occurred,  and  to  what  extent  they  could  mitigate  the  effects  of 
instability  once  it  did  occur.  Neither  the  Program  Offices  nor  the  contractors  reported  much 
success  in  avoiding  instability  or  mitigating  its  effects.  But  both  groups  were  reportedly  more 
successful  in  mitigating  negative  effects  than  avoiding  instability  in  the  first  place.  Few 
responded  that  they  were  “very  successful”  in  avoiding  or  mitigating  the  negative  effects  of 
instability.  When  efforts  to  avoid  project  instability  fail,  managers  must  alter  their  plans,  making 
choices  and  tradeoffs  based  on  perceived  objectives  and  incentives. 105 


105  The  research  on  program  instability  was  conducted  in  collaboration  with  Dr  Eric  Rebintisch  of  the  Lean 
Aerospace  Initiative.  For  more  detailed  information  please  see  his  report:  “Managing  Under  and  With  Program 
Instability”.  Lean  Aerospace  Initiative  Working  Paper. 
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Figure  11-14:  Distribution  of  Responses  of  Success  in  Mitigating  Negative  Effects  of 
Instability  (Contractor  and  Pentagon  Surveys;  Number  of  Projects  =  158) 

D.  Responses  to  Changes 

The  Pentagon  and  Program  Office  surveys  asked  respondents  to  indicate  the  likelihood 
that  cost,  performance,  and  schedule  would  change  in  response  to  unforeseen  events.  The 
responses  indicate  that  technical  performance  was  the  least  likely  to  change.  Significantly  more 
likely  to  change  were  the  total  cost  of  the  system  and  the  length  of  the  schedule.  Project 
schedule  was  reported  significantly  more  likely  to  change  than  total  cost.  This  is  consistent  with 
the  stated  project  objectives  found  in  the  planning  phase. 

Of  205  respondents,  113  indicated  that  schedule  was  more  likely  to  change  than 
performance,  while  only  46  respondents  indicated  that  performance  was  more  likely  to  change 
than  schedule.  Forty-one  indicated  that  schedule  and  performance  were  equally  likely  to  change. 
In  the  cost-schedule  tradeoff,  83  respondents  indicated  that  schedule  was  more  likely  to  change 
than  cost,  while  56  respondents  indicated  that  cost  was  more  likely  to  change  than  schedule. 
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Sixty-six  respondents  indicated  that  they  were  equally  likely  to  change.  The  statistical 
significance  of  the  difference  in  the  performance/schedule  trade-off  using  the  sign  test  is  99.99 
percent.  The  statistical  significance  of  the  difference  in  cost/schedule  tradeoffs  is  greater  than 
98.5  percent.  These  results  are  consistent  with  previously  reported  project  objectives. 

The  results  were  similar  when  separating  Pentagon  and  Program  Office  respondents.  On 
average,  Pentagon  respondents  reported  that  all  items  were  slightly  less  likely  to  change  than 
reported  by  Program  Office  respondents.  The  relative  order  of  the  factors  was  the  same,  and 
schedule  was  reported  as  the  most  likely  to  change  in  both  surveys. 
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Figure  11-15:  Likelihood  That  Cost,  Schedule,  and  Performance  Would  Change  in 
Response  to  Unforeseen  Events  to  Achieve  Project  Objectives  (Pentagon  and  Program 
Office  survey;  Number  of  Projects  =  205. 
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Figure  11-16:  Likelihood  That  Cost,  Schedule,  and  Performance  Would  Change  in 
Response  to  Unforeseen  Events  (Number  of  Projects  =  205) 


Significant  differences  appeared  between  respondents  working  under  cost-plus  and  those 
working  under  fixed-price  contracts,  but  not  in  the  relative  order  of  the  responses.  For  example, 
project  managers  of  fixed-price  contracts  were  more  likely  to  change  schedules  and  less  likely  to 
change  product  performance  than  managers  of  cost-plus  contracts.  But  managers  on  both  types  of 
contracts  reported  that  they  were  more  likely  to  change  schedules  than  to  change  performance 
requirements.  This  is  consistent  with  the  view  of  the  schedule  as  the  lowest  project  priority. 
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Figure  11-17:  Likelihood  That  Cost,  Schedule,  and  Performance  Would  Change  by  Type 
of  Contract  (Pentagon  and  Program  Office  Surveys;  Number  of  Projects=205) 


E.  Project  Changes  during  Development 

To  determine  the  amount  of  change  that  occurs  in  a  development  project,  the  Pentagon 
Survey  asked  respondents  to  indicate  to  what  extent  cost,  schedule,  and  performance 
requirements  had  changed  since  Milestone  1  or  its  equivalent.  The  amount  of  change  that  occurs 
during  the  development  phase  is  significant.  The  largest  amount  of  change  was  reported  in  the 
budget  profile  and  schedule  areas,  with  significantly  less  change  in  performance  requirements. 
This  is  consistent  with  earlier  findings.  One-third  of  all  projects  reported  a  significantly  different 
or  entirely  new  budget  profile  and  schedule. 

The  changes  in  the  budget,  schedules,  and  performance  requirements  are  not  driven  by 
external  changes  such  as  changes  in  the  threat  or  mission  concept.  The  respondents  reported 
significantly  less  change  in  service  acquisition  priorities,  mission  concept,  and  threat  or 
adversary  that  the  system  was  meant  to  address. 
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No  Change  Some  Change  Entirely  New 


Figure  11-18:  Amount  of  Change  in  Performance  Requirements,  Schedule,  and  Budget 
Profile  (Pentagon  Survey;  Number  of  Projects  =  60) 


Mean  +/-  1  SE 


Figure  11-19:  Program  Changes  since  Milestone  1  or  Program  Start  as  Reported  by 
Pentagon-Level  Respondents  (Number  of  Projects  =  59) 
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There  is  a  significant  correlation  between  the  reported  level  of  change  in  the  project 
schedule  and  the  level  of  change  in  the  budget  profile.  The  Spearman  correlation  coefficient 
between  the  two  changes  is  0.72,  indicating  a  strong  positive  link.  This  is  consistent  with  the 
earlier  finding  that  funding  instability  (changes  in  budget  profile)  is  a  primary  cause  of  schedule 
slip.  The  correlation  between  changes  in  performance  requirements  and  in  budget  profiles,  and 
between  changes  in  performance  requirements  and  in  project  schedules,  were  much  lower  (0.35 
and  0.32,  respectively),  indicating  rather  weak  links.  The  link  between  changes  in  performance 
requirements  and  changes  in  the  cost  and  schedules  was  also  weak. 


E.l  Project  Schedule  Changes  during  Development 

To  obtain  a  more  quantitative  measure  of  the  frequency  of  schedule  changes,  the  surveys 
asked  all  respondents  to  indicate  the  number  of  times  schedules  had  been  “rephased,  rebaselined, 
or  significantly  altered”  since  the  project  was  initiated.  The  responses  show  that  a  considerable 
number  of  schedule-related  changes  occur  across  many  projects.  Across  all  three  groups 
surveyed,  the  number  of  schedule  changes  averaged  more  than  two  changes  per  project.  The 
distribution  of  responses  for  the  number  of  rephases  per  project  is  shown  in  Figure  1 1-20  and  1 1- 
21— which  indicate  that  many  projects  have  been  rephased  numerous  times. 
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Figure  11-20:  Number  of  Rephases,  New  Project  Baselines,  or  Significantly  Altered 
Schedules  per  Project  from  Program  Initiation  to  Time  of  Survey  (Pentagon,  Program 
Office,  and  Contractor  Surveys;  Number  of  Projects  =  260) 
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The  responses  were  consistent  across  the  three  surveys.  Less-than-optimal  wording  on 
the  Pentagon  Survey  may  have  led  respondents  to  indicate  an  extra  rephase,  or  a  single  rephase 
when  none  occurred.106 
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Figure  11-21:  Distribution  of  Number  of  Rephases  by  Three  Different  Surveys  (Pentagon 
N=50;  Contractor  N=97;  Program  Office  N=113) 

To  determine  at  what  point  rephasing  or  establishment  of  new  baselines  occur,  the 
surveys  asked  respondents  to  indicate  whether  changes  occurred  in  the  pre-engineering 
manufacturing  development  phase  (pre-EMD),  the  engineering  manufacturing  development 
phase  (EMD),  or  the  production  phase.  On  a  project-by-project  basis,  no  systematic  and 
significant  differences  were  evident  in  the  number  of  changes  in  the  pre-EMD  phases,  the  EMD 
phase,  and  the  production  phase.  However,  45  percent  of  all  projects  were  rephased  twice  or 
more  solely  in  the  EMD  phase. 

Several  cautions  are  required  when  viewing  the  data  presented  below.  First,  the  total 
number  of  rephases  by  project  phase  exceeded  the  total  number  of  reported  rephases  per  project. 
This  indicates  that  many  of  the  rephases  affected  multiple  phases  of  a  project.  Second,  the  lower 


106  The  Pentagon  Survey  question  B.l  1  asked  “How  many  times  has  the  project  been  rebaselined,  rephased,  or 
significantly  altered  after  it  began”  and  in  the  space  for  the  answer  had  “Total  number  of  project  baselines  or  project 
plans.”  It  appeared  the  less  then  optimal  wording  may  have  caused  confusion.  The  answer  as  originally  provided  is 
used  above. 
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number  of  rephases  during  the  production  phase  may  be  affected  by  the  lower  number  of 
projects  that  have  progressed  into  the  production  phase,  not  more  stability  in  that  phase. 
Approximately  50  percent  of  the  projects  had  entered  production  at  the  time  of  the  survey. 
None  of  the  phases  seem  to  be  less  susceptible  to  schedule  changes  than  others. 


Figure  11-22:  Number  of  Project  Rephases  by  Acquisition  Phase  (Pentagon,  Program 
Office,  and  Contractor  Surveys;  Number  of  Projects  =  260) 


E.2  Contract  Changes  during  Development 

Another  measure  of  change  in  development  programs  is  the  number  of  contract  changes. 
The  Contractor  survey  asked  project  managers  to  indicate  both  the  number  of  contract  changes 
and  the  areas  that  the  changes  affected  in  each  phase.  Only  7  of  the  92  projects  responding 
reported  no  contract  changes.  The  largest  number  of  contract  changes  concerned  scope  and 
requirements,  followed  by  funding-related  changes.  Many  contract  changes  dealt  with  schedule 
and  the  number  of  deliverables. 
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Figure  11-23:  Number  of  Contract  Changes  per  Project  (Contractor  Survey;  Number  of 
Projects  =  92) 

The  type  of  contract  did  influence  the  number  and  type  of  contract  changes.  Managers  of 
fixed-price  contracts  reported  more  changes  in  the  number  of  productionitems,  funding,  and 
schedules.  Managers  of  cost-plus  contracts  reported  more  changes  relating  to  project  scope  and 
requirements.  This  is  consistent  with  the  conventional  wisdom  that  managers  can  more  easily 
change  requirements  on  cost-plus  contracts  than  on  fixed-price  contracts.  A  large  number  of 
changes  occurred  per  project  under  both  types  of  contracts,  with  each  type  averaging  nearly  50 
contract  changes  to  date. 
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Figure  11-24:  Average  Number  of  Contract  Changes  per  Project  to  Date  by  Contract  Type 
and  by  Areas  That  They  Affect  (Contractor  Survey;  Fixed-Price  N  =  36;  Cost-Plus  N  =  42) 

Only  23  percent  of  respondents  reported  no  schedule-related  contract  changes  during  the 
EMD  phase.  And  only  30  percent  reported  no  schedule-related  contract  changes  in  the 
production  phase.  Over  all  phases,  only  18  percent  of  contractor  respondents  reported  no 
schedule-related  contract  changes. 
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Figure  11-25:  Number  of  Schedule-Related  Contract  Changes  by  Project  in  the  Engineering 
Manufacturing  Development  Phase  (Contractor  Survey;  N  =  39) 
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Because  incentives  provided  by  the  Program  Office  are  based  on  adherence  to  the  current 
schedule,  not  the  original  schedule,  any  schedule  changes  affect  those  incentives.  With  three- 
quarters  of  development  efforts  changing  their  contracted  schedule,  the  few  incentives  for 
completing  projects  within  the  original  contracted  schedule  are  often  lost. 


F.  Barriers  to  Shortening  Project  Schedules 

The  Pentagon  and  Program  Office  Surveys  asked  respondents  about  the  difficulty  of 
shortening  project  schedules,  the  amount  of  time  required  to  obtain  approval  for  a  shorter  schedule, 
and  the  likelihood  of  gaining  that  approval.  The  survey  also  asked  about  the  factors  entailed  in 
deciding  whether  to  approve  shorter  project  schedules,  and  in  evaluating  the  technical  possibilities 
for  shortening  a  project. 


F.l.  Lengthening  versus  Shortening  Project  Schedules 

The  Program  Office  Survey  asked  two  questions  about  the  difficulty  of  lengthening  or 
shortening  project  schedules  by  20  percent.  Project  managers  reported  that  it  was  much  easier  to 
lengthen  a  project’s  schedule  than  to  shorten  it.  For  example,  70  percent  said  was  very  difficult 
to  shorten  schedules,  while  only  19  percent  indicated  that  it  was  very  difficult  to  lengthen  the 
schedule.  In  all,  1 1 1  project  managers  stated  that  it  was  more  difficult  to  shorten  a  program  than 
to  lengthen  it,  while  only  25  project  managers  reported  the  opposite.  Twenty-five  project 
managers  reported  that  it  was  equally  hard  to  shorten  or  lengthen  the  schedule. 
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Figure  11-26:  Ease  of  Shortening  or  Lengthening  a  Project  Schedule  by  20  Percent,  as 
Reported  by  Government  Project  Managers  (Program  Office  Survey;  Number  of  Projects 
=  146) 


In  follow-up  interviews,  both  government  and  contractor  managers  confirmed  that  it  is 
significantly  more  difficult  to  shorten  project  schedules  than  to  lengthen  them.  The  reasons  and 
incentives  for  shortening  schedules  would  therefore  have  to  be  relatively  strong  to  induce  project 
managers  to  the  attempt  to  do  so.  As  shown  earlier,  the  incentives  to  try  to  shorten  schedules  are 
not  strong.  The  reason  it  is  more  difficult  to  accelerate  than  to  lengthen  a  project  schedule  will  be 
explained  next. 
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F.2.  Critical  Factors  in  Shortening  Development  Schedules 

The  Pentagon  survey  asked  respondents  to  rate  factors  influencing  their  decision  to 
approve  or  disapprove  a  contractor’s  proposal  to  shorten  a  project  schedule  by  25  percent.  The 
question  specified  that  the  new  schedule  would  require  the  same  amount  of  development 
funding.  The  respondents  indicated  that  the  most  important  factor  in  such  a  decision  was  the 
ability  to  change  the  production  funding  profile,  or  in  what  year  the  money  was  available  to  be 
spent.  The  next-highest  consideration  was  changing  the  development  funding  profile.  The 
lowest-rated  item  was  obtaining  the  user’s  approval  to  accelerate  the  schedule.  Figure  1 1-27 
shows  the  number  of  responses  indicating  that  a  specific  factor  would  be  either  critical  or  a  non 
factor  in  deciding  whether  to  approve  a  schedule  reduction.  Some  factors  such  as  training  and 
logistics  issues  did  not  play  a  significant  role,  in  the  view  of  respondents.  The  primary  drivers 
appeared  to  be  funding  issues  and  the  Program  Office’s  ability  to  meet  the  new  expectation  for 
budget  and  schedule. 
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Figure  11-27:  Factors  Critical  and  Not  Important  in  Decisions  to  Approve  or  Disapprove  a 
Contractor  Proposal  to  Shorten  Project  Schedule  25%  (Pentagon  Survey;  Number  of 
Projects  =  48) 


F.3.  Time  Entailed  in  Decisions  to  Reduce  Project  Schedules 

To  identify  some  of  the  barriers  to  shortening  project  schedules,  the  Pentagon  and 
Program  Office  surveys  asked  how  long  respondents  estimated  it  would  take  to  approve  a 
contractor’s  proposal  to  shorten  the  project  schedule  by  25  percent,  given  with  the  same  amount 
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of  total  funding.  The  average  time  required  to  obtain  either  approval  or  disapproval  was  reported 
to  be  6  months.  While  many  projects  indicated  that  a  decision  could  be  reached  in  3  months  or 
less,  many  other  projects  indicated  that  it  could  take  significantly  longer. 


Expected  Time  For  Approval  (Months) 

Figure  11-28:  Estimated  Time  to  Approve  or  Disapprove  a  Proposed  Reduction  in  a 
Project’s  Schedule,  Given  the  Same  Amount  of  Total  Funding  (Pentagon  and  Program 
Office  Surveys;  Number  of  Projects  =  180) 


F.4.  Probability  that  a  Schedule  Reduction  Will  Be  Approved 

The  Pentagon  survey  also  asked  respondents  to  specify  the  probability  that  a  contractor 
proposal  to  reduce  project  schedule  by  25  percent  would  be  approved,  given  the  same  total 
funding.  The  average  of  the  reported  probability  was  49  percent,  but  that  figure  does  not  tell  the 
entire  story.  The  largest  number  of  managers  reported  that  a  proposal  to  shorten  the  schedule 
would  not  be  considered  let  alone  approved,  and  the  second-largest  number  stated  that  such  a 
proposal  stood  only  a  50  percent  chance  of  being  approved.  Only  8  of  the  41  projects  responding 
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indicated  a  90  percent  or  greater  chance  that  a  proposal  would  be  approved,  and  only  15  of  41 
indicated  that  it  stood  a  60  percent  or  greater  chance  of  approval. 


Figure  11-29:  Program  Element  Monitors’  Estimate  of  the  Probability  That  a  Proposed 
Schedule  Reduction  Would  Be  Approved,  Assuming  the  Same  Total  Funding  (Pentagon 
Survey;  Number  of  Projects  =  41) 


The  Pentagon  survey  also  asked  respondents  to  note  the  factors  they  primarily  consider  in 
deciding  whether  to  approve  of  a  schedule  reduction  of  25  percent.  The  primary  consideration 
most  often  cited  was  the  availability  of  funding-named  in  55  percent  of  responses.  The  ability 
to  develop  the  product  and  to  accept  program  risk  was  the  primary  consideration  in  29  percent  of 
the  projects.  Operational  considerations  were  mentioned  by  16  percent  of  respondents.  These 
results  are  supported  by  another  question  revealing  that  schedules  for  77  percent  of  all  projects 
are  limited  by  funding,  not  by  technology  or  engineering-related  aspects. 
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Summary  of  the  Analysis  of  Development  Schedules 

Changes  in  annual  funding  levels,  technical  problems,  and  revisions  in  user  requirements 
most  often  drive  changes  in  development  plans.  The  project  schedule  is  the  most  likely  aspect  to 
change  in  response  to  these  unforeseen  events.  Development  projects  average  two  significant 
schedule  rephases.  When  schedules  change,  they  are  relatively  easy  to  lengthen  and  very 
difficult  to  shorten. 

The  primary  concern  when  shortening  project  schedules  is  the  availability  of  production 
and  development  funding.  Program  element  monitors  estimated  that  a  decision  on  a  contractor 
proposal  to  shorten  project  schedule  by  25  percent  would  take  an  average  of  3  months  and  as 
long  as  3  years,  even  if  such  a  change  would  not  require  an  increase  in  total  funding.  Program 
element  monitors’  estimate  of  the  probability  of  obtaining  approval  to  shorten  a  project’s 
schedule  by  25  percent  was  50  percent,  even  without  any  increase  in  total  funding. 

One  result  of  schedule-related  incentives  was  that  development  efforts  usually  either  slip 
or  end  exactly  on-schedule.  Over  39  percent  of  all  projects  finished  within  3  months  of  their 
schedule,  and  30  percent  were  exactly  on  schedule.  Less  than  5  percent  of  projects  finished 
ahead  of  schedule.  Respondents  attributed  the  primary  cause  of  schedule  slip  to  funding 
instability.  Only  20  percent  of  all  projects,  and  30  percent  of  projects  reporting  schedule  slips, 
named  technical  problems  as  the  primarily  reasons  for  schedule  slips.  This  further  indicates  that 
technical  requirements  are  not  the  primary  driver  in  a  majority  of  development  projects. 

Program  Office  and  Pentagon  respondents  indicated  that  significant  reductions  are  possible  if 
schedules  are  given  significant  priority  and  required  funding  is  made  available.  On  average,  the 
Program  Office  managers  estimated  that,  if  needed,  projects  could  be  completed  in  roughly  half  the 
remaining  schedule.  Pentagon  respondents  estimated  that  the  time  required  to  develop  the  project 
without  resource  constraints  was  73  percent  of  the  planned  schedule. 

The  influence  of  the  planned  and  contracted  schedule  on  the  length  of  the  development 
period  is  evident  in  the  fact  that  few  projects  are  ever  completed  ahead  of  the  initial  schedule. 

The  schedule  established  early  in  the  project’s  planning  stage  essentially  sets  the  minimum  time 
it  will  take  to  complete  the  project.  As  shown  in  earlier  chapters,  the  initial  project  schedule  is 
not  based  primarily  on  development-related  requirements  but  rather  on  the  expected  availability 
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of  funding.  Once  the  initial  schedule  is  established,  incentives  throughout  the  contracting  and 
development  phases  encourage  managers  to  meet  the  planned  schedule  and  not  reduce  it.  The 
outcomes  of  project  schedules  show  that  this  is  the  case. 


Part  4 

Conclusions ,  Observations ,  and 
Recommendations 


Part  4:  Overview 

Part  4  draws  conclusions  from  the  research  results  on  the  project  level  and  makes 
observations  regarding  the  overall  development  process.  These  lead  to  specific  policy 
recommendations  on  the  steps  considered  necessary  to  shorten  development  times  for  new 
military  systems. 

Chapter  12  discusses  the  schedule  development  process  and  draws  specific  conclusions 
on  that  process  based  on  the  research.  Chapter  13  makes  observations  on  the  overall 
development  processes  that  can  be  drawn  from  the  research  and  identifies  key  problem  areas  that 
must  be  addressed  to  reduce  development  time.  Chapter  14  presents  specific  recommendations 
for  addressing  those  problems.  Chapter  14  concludes  with  a  discussion  of  the  potential  changes 
in  weapon  development  strategies  made  possible  by  dramatically  shorter  development  times. 
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Chapter  12 


Conclusions  Regarding  the  Process  Used  to 
Develop  and  Execute  Project  Schedules 


Data  from  the  three  surveys  and  discussion  of  the  process  used  to  develop  and 
execute  the  project  schedule  leads  to  conclusions  about  the  impact  of  the  schedule  on  the 
eventual  outcome.  These  conclusions  are  based  on  information  from  317  separate 
projects  as  well  as  interviews  conducted  at  the  Pentagon,  Program  Office,  and  the 
contractor  levels.  While  these  results  do  not  represent  what  occurs  on  a  specific  project 
under  specific  circumstances,  they  do  represent  the  processes  used  across  the  great 
majority  of  Air  Force  development  projects. 

These  conclusions  can  yield  specific  recommendations  on  how  to  change  the 
schedule  development  process  and  the  organizational  focus  to  shorten  the  time  required  to 
develop  new  projects. 


307 


308  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Conclusion  1:  Short  development  times  are  not  a  significant  priority  for 
military  development  projects. 

Despite  the  fact  that  80  percent  of  Pentagon  program  element  monitors  report  that 
users  want  systems  “as  soon  as  possible,”  and  that  70  percent  of  projects  are  needed  to 
meet  current  operational  deficiencies,  (Figure  8-2  and  8-3),  shortened  schedules  most 
often  rank  fourth  of  four  project  objectives  (Figure  8-5).  This  goal  ranked  significantly 
lower  than  bettering  project  performance  and  lowering  acquisition  cost.  As  shown  in 
Figure  8-6,  only  15  percent  of  Program  Office  project  managers  and  Pentagon  program 
element  monitors  ranked  short  schedule  very  important;  over  half  of  the  respondents 
indicated  that  a  short  acquisition  cycle  was  only  “somewhat  important”  or  “not 
important.” 

The  lack  of  priority  placed  on  schedules  can  also  be  seen  in  the  contracting 
process  and  the  criteria  used  to  select  the  contractor.  As  shown  in  Figure  9-9, 65  percent 
of  project  managers  and  program  element  monitors  indicated  that  development  time  was 
only  “somewhat  important”  or  “not  important”  as  a  criterion  for  selecting  a  contractor. 
Figure  9-10  reveals  that  the  vast  majority  of  contractors  indicated  they  had  no  incentive  to 
bid  a  shorter  schedule. 

The  results  are  seen  in  projects’  schedule-related  results.  Figure  1 1-1  and  Figure 
1 1-4  reveal  that  very  few  projects  are  completed  ahead  of  schedule— despite  the  estimate 
by  project  managers,  shown  in  Figure  1 1-8,  that  the  average  project  could  be  completed 
in  half  the  time  remaining  and  the  estimate  by  the  program  element  monitors  that  the 
programs  could  be  completed  in  65%  of  the  original  planned  time  as  shown  in  Figure  11- 
9. 


From  these  results  and  analysis  across  the  planning,  contracting,  and  development 
phases,  I  conclude  that  shortening  development  schedules  is  not  seen  as  a  high  project 
priority  within  the  current  Air  Force  development  system. 


12.  Conclusions  Regarding  the  Process  Used  to  Develop  and  Execute  Project  Scehdules 
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Conclusion  2:  The  initial  project  schedule  has  significant  influence  on 
the  minimum  time  taken  to  develop  a  project. 

The  effect  of  the  initial  project  schedule  can  be  seen  in  the  contracting  and 
development  phases.  As  shown  in  Figure  9-2,  80  percent  of  requests  for  proposal  specify 
an  expected  project  schedule.  As  revealed  in  Figure  9-5,  the  Program  Office’s  expected 
schedule  becomes  the  primary  input  to  the  contractor’s  proposed  schedule.  As  shown  in 
Figure  9-6, 66  percent  of  contractor  respondents  indicated  that  the  initial  schedule  was  the 
sole  determinate,  or  nearly  the  sole  determinate,  of  their  proposed  schedules.  Figure  9-10 
shows  that  contractors  say  they  have  no  incentive  to  bid  a  schedule  different  from  the  one 
the  Program  Office  expects.  Figure  9-12  and  9-13  shows  that  a  large  majority  of 
schedules  proposed  by  contractors  exactly  match  the  government’s  expected  schedule. 
These  proposals  form  the  basis  for  development  contracts. 

In  the  development  phase,  the  objectives  of  the  Pentagon,  Program  Offices,  and 
contractors  are  primarily  to  meet  cost,  schedule,  and  performance  goals  and  not  to  exceed 
them.  Figure  10-1 1  shows  few  incentives  for  Program  Offices  to  shorten  schedules, 
while  Figures  10-16  and  10-17  show  that  the  Program  Offices  provide  few  incentives  to 
contractors  to  meet  or  shorten  project  schedules.  Figure  10-18  shows  that  the  contractors, 
in  turn,  feel  little  incentive  to  exceed  project  objectives. 

Chapter  1 1  similarly  revealed  little  evidence  of  any  incentives  to  exceed,  as 
opposed  to  meeting,  a  project’s  cost,  schedule,  and  performance  goals  at  the  Pentagon, 
Program  Office,  or  contractor  levels.  As  shown  in  Figure  1 1-1, 39  percent  of  surveyed 
projects  are  within  3  months  of  their  initial  schedule.  Very  few-less  than  5  percent- 
report  being  ahead  of  schedule.  As  illustrated  in  Figure  1 1-4,  these  results  are  consistent 
with  information  in  the  RAND  database  drawn  from  selected  acquisition  reports  for  all 
major  defense  projects. 

From  these  data  showing  the  clear  links  among  the  initial  project  schedule,  the 
contracted  schedule,  and  the  resulting  development  time,  I  conclude  that  the  initial  project 
schedule  has  a  significant  impact  on  the  minimum  time  it  takes  to  develop  a  project. 
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Conclusion  3:  Initial  project  schedules  are  determined  primarily  by 
expected  development  and  production  funding  rather  than  development- 
related  requirements. 

This  conclusion  is  supported  by  results  from  many  questions  in  the  three  surveys. 
Figure  8-13  and  figures  8-14  show  that  the  user’s  desired  date,  expected  development 
funding,  and  expected  production  funding  were  the  most  significant  influences  on  the 
length  of  a  project’s  initial  schedule  at  both  the  Pentagon  and  Program  Office.  But  the 
process  for  allocating  resources  described  in  Chapter  3  and  significant  discussions 
outlined  in  Chaopter  8  reveal  that  the  user’s  desired  date  is  determined  primarily  by  the 
availability  of  funding. 

Figure  8-16  shows  that  77  percent  of  respondents  report  that  their  schedules  are 
limited  by  funding,  not  technology  or  engineering.  Tables  8-8  through  8-10,  and  Figures 
8-17  through  8-20,  show  that  expected  development  funding  rated  more  important  on 
more  projects  than  engineering  requirements,  technology  development,  and  development 
of  the  manufacturing  process  in  every  category,  including  size  of  project  (ACAT  I,  II,  III 
levels),  type  of  project  (aircraft,  spacecraft,  electronic,  missiles,  software,  or  other),  and 
amount  of  technological  advance  (revolutionary  product,  new  generation,  or  incremental 
improvement). 

As  shown  in  Figure  8-22,  the  information  most  influencing  development  of  the 
initial  schedule  is  expected  development  funding  and  expert  judgment.  Contractor 
proposals,  historical  similar  projects,  bottom-up  schedule  development,  and  parametric 
modeling  were  all  reported  to  have  less  impact  on  development  of  the  initial  schedule. 
Comparable  commercial  development  efforts  were  rated  significantly  lower  than  any 
other  factor. 

Figures  11-5  and  11-6  underscore  the  fact  that  a  majority  of  respondents  report  the 
primary  cause  of  schedule  slips  to  be  funding  instability  and  changes  in  requirements, 
not  technical  problems.  If  technical  development  issues  were  the  limiting  factor,  one 
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would  expect  more  than  20  percent  of  projects  to  report  technical  problems  as  the  primary 
cause  of  schedule  delays  (see  Table  11-15). 

Based  on  this  information,  I  conclude  that  the  length  of  the  initial  development 
schedule  is  primarily  determined  by  the  amount  of  expected  funding  for  a  project,  and  not 
development-related  requirements. 


Conclusion  4:  There  are  few  incentives  at  any  level  to  reduce  project 
schedules  and  development  time. 

This  is  evident  from  responses  regarding  project  objectives  in  the  planning  phase, 
incentives  during  the  proposal  stage,  and  incentives  at  the  Pentagon,  Program  Office  and 
contractor  levels  during  development. 

The  incentives  associated  with  a  project  begin  with  its  objectives.  As  shown  in 
Figure  8-5,  Figure  10-5  and  Figure  10-8,  short  schedules  are  the  lowest  priority  compared 
with  increasing  performance  and  cutting  acquisitions  and  operating  costs,  in  both  the 
Pentagon  and  the  Program  Offices. 

Figure  9-9  shows  that  65  percent  of  project  managers  and  program  element 
monitors  did  not  report  development  time  as  an  important  criterion  in  selecting 
contractors.  In  Figure  9-5,  contractors  reported  that  the  government-proposed  schedule 
was  the  primary  consideration,  and  in  many  cases  the  sole  determinate,  in  their  proposed 
schedules.  Figure  9-10  shows  that  the  contractors  felt  “no  incentive”  to  bid  a  schedule 
different  from  the  government’s  expected  schedule,  and  Figure  9-11  shows  that  this  was 
consistent  for  both  competitive  and  sole-source  contracts.  Figure  9-12  and  Figure  9-13 
show  that  the  vast  majority  of  proposals  exactly  match  the  government’s  expected 
schedule. 

The  incentives  during  the  development  stage  to  reduce  development  time  are  also 
reported  to  be  small.  As  shown  in  Figure  10-5,  Pentagon  respondents  indicated  that  the 
first  and  second  goal  of  a  project  is  to  increase  performance  and  lower  cost.  Shortening 
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schedule  was  reported  as  the  fourth  of  four  objectives  at  the  Pentagon  and  Program  Office 
levels.  As  shown  in  Figure  10-10  the  incentives  provided  to  the  Program  Office 
primarily  center  on  meeting  project  objectives  and  not  exceeding  them.  As  shown  in 
Figure  10-1 1,  in  over  half  the  projects  the  Program  Office  project  managers  are  reported 
to  have  no  or  even  negative  incentives  to  shorten  cycle  time. 

Contractors’  incentives  to  shorten  cycle  times  are  similarly  weak.  As  shown  in 
Figures  10-16  and  10-17,  Program  Offices  reported  that  in  a  vast  majority  of  projects, 
contracts  do  not  include  any  financial  incentive  for  on-time  or  early  completion  of  the 
project.  In  Figure  10-18  contractors  reported  that  they  feel  no  incentive  to  exceed  any 
project  goals.  Figure  10-20  also  shows  that  73  percent  of  contractors  reported  no  or 
negative  customer-provided  incentives  to  shorten  project  schedules.  In  only  18  percent  of 
the  cases  did  contractors  indicate  that  the  Program  Office  provided  more  than  a  slight 
incentive  to  shorten  schedule.  Figures  10-21  and  10-22  show  that  the  type  of  contract 
(Fix  Priced  or  Cost  Plus)  or  the  presence  of  competition  had  little  or  no  effect  on  the 
incentive  to  reduce  schedule  provided  by  the  government.  As  shown  in  Figure  10-28, 
contractors  involved  in  fixed-price  contracts  did  report  slightly  higher  levels  of  overall 
incentives  than  those  on  cost-plus  contracts. 

Based  on  this  information,  I  conclude  that  there  is  a  lack  of  significant  incentives 
at  all  levels  to  shorten  project  schedules  and  development  cycle  times. 
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Conclusion  5:  There  are  significant  structural,  cultural,  and 
organizational  barriers  in  the  schedule  development  process  that  inhibit 
shorter  schedules. 

Once  a  project  is  under  way,  project  managers  consider  shortening  the  schedule 
very  difficult,  as  Figure  11-26  shows.  Figure  11-27  similarly  reveals  that  program 
element  monitors  believe  the  critical  factor  in  deciding  whether  to  approve  a  proposal  to 
accelerate  a  project  is  the  ability  to  change  production  and  development  funding  profiles. 

As  shown  in  Chapter  3,  the  development  funding  process  is  elaborate,  cumbersome,  and 
difficult  to  change  late  in  the  budgetary  process.  Figure  1 1-28  shows  that  Program 
Offices  and  program  element  monitors  estimate  that  6  months  would  be  required  for  a 
decision  on  a  contractor’s  proposal  to  shorten  a  project’s  schedule  by  25  percent,  even 
without  any  increase  in  total  project  funding.  As  shown  in  Figure  11-29,  program 
element  monitors  estimated  the  average  probability  of  approval  for  a  proposal  to  shorten 
schedules  without  increasing  total  program  funding  at  49  percent.  The  largest  number 
reporting  the  chance  of  approval  at  0  percent.  The  primary  consideration  was  the 
availability  of  changing  funding.  This  is  consistent  with  Figure  8-16  that  shows  77 
percent  of  the  projects  report  to  be  funding  limited  as  opposed  to  technology  or 
engineering  limited. 

The  cultural  barriers  begin  with  the  low  priority  placed  on  project  schedules:  they 
are  seen  as  dictated  by  funding-related  decisions  and  technical  requirements.  Schedule- 
related  feedback  from  Program  Offices  and  defense  contractors  is  limited  primarily  to 
when  the  schedule  cannot  be  met.  Once  schedules  and  funding  are  determined,  Program 
Offices  do  not  have  the  authority  or  ability  to  adjust  allocated  resources  to  match  the 
optimal  project  schedules.  The  result  is  that  available  funding  drives  the  schedule,  but 
the  possible  schedule  does  not  appear  to  drive  funding  decisions. 

Based  on  these  factors  I  conclude  that  there  are  significant  structural,  cultural,  and 
organizational  barriers  to  shortening  cycle  times.  Within  the  schedule  development 
process,  significant  hurdles  block  effective  feedback  on  schedule-related  issues. 
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B.  Review  of  the  Data  and  Conclusions 

The  data  and  conclusions  from  this  research  were  presented  to  a  number  of 
organizations  as  an  aid  to  interpreting  their  validity.  Representatives  from  the  acquisition 
reform  offices  within  the  Pentagon,  the  Aeronautical  Systems  Center,  and  the  Electronic 
Systems  Center  agree  significantly  with  the  information. 

The  results  and  conclusions  were  also  presented  to  representatives  from 
government  organizations,  program  offices,  and  representatives  from  the  major  defense 
aerospace  companies  at  the  Lean  Aircraft  Initiative  Plenary  Workshop  in  October  1997. 
Following  the  presentation,  each  group  was  asked  to  review  the  data  from  one  of  the 
development  phases  in  detail  and  then  asked  to  report,  through  a  written  survey,  whether 
the  results  and  conclusions  of  the  research  matched  their  personal  experience.  The  results 
were  surprising  only  in  their  near-unanimity:  among  90  responses,  88  people  reported 
that  the  data  matched  their  personal  experiences.  Only  2  reported  being  neutral  on 
whether  the  data  match  their  experiences.  No  one  reported  that  the  data  and  conclusions 
were  contrary  to  their  personal  experiences. 
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Figure  12-1:  Responses  from  the  Lean  Aerospace  Initiative  Plenary  Workshop  as  to 
Whether  the  Data  and  Conclusions  Match  Their  Personal  Experience  LAI  Plenary 
Session  Hartford  CT.  8  October  97  (Number  of  Respondents  =  90) 


Summary 

Based  on  the  analysis  of  survey  data  on  317  projects  drawn  from  a  wide  cross 
section  of  current  development  efforts,  and  the  strong  acceptance  of  the  results  and 
conclusions  of  the  research  by  those  experienced  in  the  defense  acquisition  processes,  the 
data  and  conclusions  must  be  accepted  as  accurately  representing  the  processes  and 
procedures  used  to  develop  and  execute  project  schedules.  The  data  and  conclusions, 
while  not  representing  what  occurs  on  a  specific  project  under  specific  circumstances,  do 
represent  the  general  processes  employed  across  the  great  majority  of  Air  Force 
development  projects.  The  conclusions  on  the  schedule  development  process  and  its 
impact  on  development  time  lead  to  specific  observations  and  recommendations  on  how 
to  change  that  process  and  organizational  focus  to  shorten  development  time. 


Chapter  13 

Observations  on  the  Overall  Product 
Development  Process _ 


The  overall  observations  of  the  defense  product  development  system  focus  on  four  specific 
areas:  a  lack  of  clear  and  consistent  leadership  on  schedule  reduction;  the  lack  of  schedule-based 
information  from  which  to  make  decisions;  the  lack  of  schedule-based  incentives  at  all  levels;  and  the 
overriding  influence  of  the  funding  process.  Within  each  area,  specific  observations  about  the 
development  system  will  lead  to  specific  recommendations  addressed  in  the  next  chapter. 


1.  Lack  of  Leadership  on  Cycle  Time 

Observation  1A:  There  has  been  little  leadership  or  emphasis  on  reducing  the  time  to 
develop  new  products,  and  shortening  development  time  is  not  seen  as  an  important 
priority. 

Development  organizations  focus  on  increasing  performance,  reducing  cost,  and  staying  within 
the  allocated  budget.  Schedule  is  most  often  seen  as  an  outcome  of  these  other  objectives  and  not 
something  to  be  actively  managed.  The  reason  is  that  there  is  no  consensus  on  or  appreciation  of  the 
impact  of  cycle  time  within  the  development  community.  The  survey  and  interviews  indicate  this  at  all 
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levels,  from  senior  Air  Force  and  DoD  leadership  down  to  project  engineers.  The  most  prevalent  view  at 
each  level  is  that  shortening  schedules  is  a  low  priority. 

I  believe  the  reason  for  this  view  is  the  significant  lack  of  reliable  information  on  which  to  base 
judgments.  Those  interviewed  view  cycle  time  as  it  affects  their  job  and  their  organizational  goals;  their 
experiences  and  the  limitations  they  face  do  not  allow  them  see  how  long  cycle  times  affect  the 
development  process,  cost  of  systems,  or  military  capability  as  a  whole.  What  is  clear  is  that  no  one  has 
made  a  clear  and  compelling  case  for  reducing  cycle  time  for  military  projects.  No  rationale  has  been 
presented  connecting  the  issue  of  cycle  time  to  warfighters  capabilities.  No  case  has  been  made  on  the 
effects  of  long  cycle  time  on  the  budget.  No  case  has  been  made  on  the  effects  of  long  development 
times  on  the  acquisition  community.  Without  a  consensus  among  key  personnel  on  the  importance  of 
reducing  development  times,  little  progress  will  be  made  on  reducing  them. 


Observation  IB:  There  is  no  visible  or  widely  held  goal  of  reducing  cycle  time. 

Throughout  this  research  effort,  no  goals  related  to  reducing  cycle  time  were  found  to  be 
accepted  or  used  across  any  of  the  services  or  development  organizations.  Significant  goals  are 
associated  with  increasing  performance  and  reducing  cost-survey  respondents  rate  them  as  the 
first  and  second  priority. 


Observation  1C:  There  is  no  advocate  for  reducing  cycle  time. 

Users,  who  one  would  expect  to  be  the  strongest  advocates  for  shortening  development 
time,  have  not  made  doing  so  a  priority.  Instead  users  have  traditionally  stressed  better  system 
performance  and  improvements  in  reliability,  maintainability,  and  supportability,  and  focused  on 
starting  additional  projects  rather  than  on  reducing  development  times.  The  acquisition 
community  has  traditionally  advocated  new  technology  and  systems,  although  it  is  now  focusing 
on  cutting  acquisition  and  life-cycle  costs  as  part  of  reform  efforts.  The  budget  and  oversight 
communities  have  also  stressed  annual  cost  reductions  and  accountability.  No  organization  was 
found  to  be  a  strong  advocate,  or  to  have  responsibility,  for  shortening  development  times. 
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2.  Lack  of  Schedule-Related  Information  and  Tools 

While  significant  effort  is  placed  on  determining  the  required  cost  of  a  proposed  project, 
relatively  little  emphasis  has  been  focused  on  accurately  determining  the  required  schedule.  As 
shown  by  the  research,  project  schedules  are  based  to  a  greater  extent  on  expected  funding; 
Relatively  little  effort  by  the  Pentagon,  Program  Offices,  or  defense  contractors  is  devoted  to 
analyzing  the  appropriateness  of  schedules  vis-a-vis  development-related  requirements.  This 
absence  of  rigor  stems  from  the  overall  lack  of  emphasis  on  schedules,  the  lack  reliable 
information,  and  the  lack  of  effective  tools  for  analyzing  schedules. 


Observation  2A:  Schedules  are  based  on  factors  other  than  the  actual  time  required  to 
develop  a  project.  Once  established,  these  schedules  determine  the  minimum  time  it  will 
take  to  develop  a  product. 

Project  schedules  are  not  based  primarily  on  the  time  required  to  develop  the  project,  as 
determined  by  technological  and  engineering  requirements  and  the  manufacturing  process.. 
Instead,  schedules  are  determined  primarily  by  expected  development  and  production  funding. 
This  was  found  consistently  across  all  sizes  and  types  of  projects.  These  schedules  determine  the 
minimum  time  that  it  takes  to  develop  a  project,  since  very  few  are  ever  completed  ahead  of 
schedule.  Once  established,  the  initial  project  schedule  becomes  a  self-fulfilling  prophecy. 


Observation  2B:  Little  rigorous  schedule-based  information  is  available  to  plan,  manage, 
or  evaluate  timetables  for  development  projects. 

This  research  entailed  significant  efforts  to  locate  detailed  schedule-based  information, 
but  little  such  information  other  than  the  resulting  schedules  was  found  for  most  projects.  The 
only  schedule  information  generally  available  is  based  on  acquisition  milestones  or  major  events, 
which  do  not  evaluate  the  appropriateness  of  the  schedules  or  allow  comparisons  among  projects. 
The  scheduling  rationale  and  possible  alternatives  are  typically  not  documented.  Several  other 
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researchers  have  made  similar  observations.107  The  schedule  information  that  is  available 
concerns  funding.  Thus,  schedule  becomes  the  output,  or  dependent  variable,  of  the  cost  and 
performance  of  the  project. 

The  lack  of  schedule-related  information  makes  it  difficult  to  evaluate  the  feasibility,  risk,  and 
appropriateness  of  alternatives.  With  little  supporting  data  and  analysis,  discussion  and  decisions 
are  based  mainly  on  people’s  limited  experiences  and  judgment,  and  on  elaborate  cost  models. 


Observation  2C:  There  are  no  effective,  accepted,  or  widely  used  schedule-estimating  tools 
for  military  development  projects.  The  lack  of  such  tools  limits  the  ability  to  make 
decisions  based  on  good  information. 

The  lack  of  rigor  in  used  to  develop  initial  schedules  can  be  attributed  partly  to  the  lack  of 
available  and  easy-to-use  schedule-estimating  tools.  The  research  turned  up  no  scheduling 
guides  or  standard  references  for  devising  a  breakdown  of  activities  and  their  expected  duration. 
Tools  dependent  on  information  from  other  projects,  such  as  bottom-up  schedule  development 
(based  on  engineering  requirements),  comparison  with  historically  similar  programs,  and 
parametric  modeling,  are  not  used.  Comparisons  with  other  programs  are  difficult  because  very 
limited  schedule-related  information  is  available. 

The  surveys  indicated  that  no  project  had  made  significant  use  of  schedule-estimating 
tools  or  parametric  models.  Most  rely  instead  on  rudimentary  Gantt  and  milestone  charts. 
Though  many  projects  reported  “occasionally”  using  of  critical  path  management  tools  or  PERT, 
less  than  a  third  (40  of  126)  reported  extensive  use  of  either  in  developing  the  initial  project 
schedule.  No  other  scheduling  tools  were  found  to  be  widely  used  to  forecast  development  time 
based  on  the  project’s  development-related  requirements. 

The  lack  of  emphasis  on  scheduling  tools  stands  in  stark  contrast  to  the  effort  placed  on 
estimating  costs.  Several  major  organizations  at  both  the  DoD  and  the  service  levels  have  been 
established  to  develop  and  maintain  detailed  cost  models,  which  are  used  to  ensure  that  a 
development  effort  can  be  accomplished  within  the  allocated  budget.  The  cost  models  do  not 
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estimate  project  schedules  or  the  cost  of  development  time.  Based  on  discussions  with  cost 
analysts  in  the  DoD  and  each  of  the  three  services,  I  conclude  that  these  models  are  not 
significantly  affected  by  the  length  of  the  project  schedule,  except  in  a  few  minor  areas.  People 
in  charge  of  such  models  estimated  that  the  models  would  predict  that  doubling  a  project 
schedule  would  increase  a  project’s  total  cost  by  5  percent,  owing  to  higher  managerial  costs  to 
the  contractors.108  To  achieve  such  dubious  results,  the  cost  models  ignore  many  important 
considerations. 

One  such  consideration  is  the  fact  that  the  longer  a  project  lasts,  the  more  its  costs  grow. 
Among  the  projects  surveyed  he  average  cost  growth  owing  to  funding  instability  and 
requirement  changes  alone  was  4.6  percent  per  year.  The  reported  total  annual  cost  growth  was 
6.4  percent.  The  Rand  database  shows  that  annual  cost  growth  for  major  defense  acquisition 
programs  averages  is  5.2  percent.109 

The  models  also  fail  to  include  the  opportunity  costs  of  delaying  a  system’s  availability. 
Because  these  models  use  costs  largely  as  a  fixed  input  and  not  an  output,  they  are  not  capable  of 
estimating  required  or  optimal  schedules,  or  of  determining  the  impact  of  various  schedules  on 
cost. 

Previous  attempts  to  identify  relationships  based  on  project  characteristics  have  proven 
largely  unsuccessful  and  have  been  hampered  by  lack  of  data,  and  have  not  had  an  impact  on 
development  times.  The  data  that  are  available  do  not  separate  the  effect  of  funding  limitations 
from  technical  limitations.  Most  schedule-related  efforts  have  tried  to  identify  the  causes  of 
schedule  slip,  not  the  adequacy  of  the  planned  schedule.  Without  detailed  schedule-related 
information  and  tools  to  analyze  it,  scheduling  decisions  will  continue  to  be  unsupported.  And 
schedule-based  arguments  are  not  likely  to  have  the  impact  of  other  cost-based  arguments  backed 
by  analysis. 


108  Interviews  with  representatives  from  the  DoD  (Cost  Analysis  Improvement  Group,  CAIG)  and  the  various  service 
cost  estimating  organizations  were  conducted  during  the  spring  of  1998  at  the  Pentagon. 

109  Calculated  from  the  Rand  SAR  database  from  the  development  cost  estimate  and  the  actual  or  current  projected 
cost  for  72  MDAP  programs. 
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Observation  2D:  Contractors  base  their  proposed  schedules  on  the  government’s  expected 
schedule,  not  on  analysis  of  the  actual  time  required  to  develop  the  product 

Not  only  did  contractors  propose  development  plans  to  meet  the  Program  Office’s 
expected  schedule,  but  those  that  did  analyze  the  required  schedule  reported  that  it  had  less  than  a 
moderate  impact  on  their  proposed  schedule.  Contractors  did  not  report  using  any  effective 
schedule  development  and  analysis  tools,  nor  were  any  observed. 

Bidding  a  different  schedule  than  that  calculated  by  the  Program  Office  is  not  seen  as  a 
winning  strategy.  Program  Office  project  managers  stated  that  different  schedules  are  presumed 
to  carry  higher  risk  or  be  non-responsive.  Without  detailed  information  on  the  contractor’s 
proposed  schedule  and  the  ability  to  analyze  it  effectively,  there  is  no  way  to  refute  this 
assumption. 


3.  Lack  of  Schedule-Based  Incentives 

The  surveys  uncovered  few  incentives  associated  with  reducing  development  times  at  any 
level  of  the  development  process,  at  the  Pentagon,  Program  Offices,  or  contractor  levels.  The 
few  incentives  that  do  exist  are  associated  with  meeting  but  not  beating  project  cost,  schedule, 
and  performance  goals. 


Observation  3A:  There  are  no  effective  incentives  for  reducing  development  time  at  either 
the  Program  Office  or  the  Pentagon  levels. 

The  lack  of  schedule-based  incentives  stems  partly  from  the  lack  of  priority  placed  on 
development  time  and  shortening  project  schedules.  No  metric  or  measurement  is  used  to  track 
schedule  performance  across  the  range  of  projects  in  development.  Few  visible  metrics  are  used 
to  track  a  specific  project’s  schedule  performance  other  than  its  current,  though  often  changed, 
plans.  Project  slips  are  viewed  as  based  on  budget  cuts  or  technical  problems. 

Project  managers  are  apparently  not  held  accountable  for  schedule  slips,  reporting  that 
schedule  performance  had  little  effect  on  their  performance  rating  and  promotion  potential.  One 
reason  there  is  little  schedule-related  incentive  is  the  high  turnover  rate  for  Pentagon  p  element 
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monitors  and  Program  Office  project  managers.  The  average  program  element  monitor  has  held 
that  position  for  fewer  than  15  months,  the  average  program  manager  for  fewer  than  18  months. 
These  tenures  are  short  compared  with  the  average  75-month  development  time  for  surveyed 
projects.  High  turnover  means  that  accountability  for  schedule  performance  is  spread  among  a 
number  of  people,  making  it  difficult  to  hold  anyone  responsible.  The  schedule  incentives  that 
project  managers  do  report  are  based  on  meeting  the  planned  schedules. 


Observation  3B:  There  are  no  effective  incentives  for  contractors  to  reduce  development 
times. 

The  strong  incentives  that  companies  report  during  the  contracting  phase  are  to  match  the 
government’s  expected  schedule  and  not  to  change  it.  The  result  is  that  nearly  all  proposals 
match  exactly  the  government  schedule.  Similarly,  contractors  report  few  or  no  incentives  to 
beat  schedules  as  a  project  proceeds.  The  schedule-based  incentives  that  do  exist  are  based  on 
meeting  the  schedule. 

Observation  3C:  No  effective  measures  are  used  to  evaluate  schedule  performance  across 
different  organizations,  or  within  the  development  process  as  a  whole. 

The  lack  of  schedule-related  incentives  appears  to  stem  partly  from  the  conspicuous 
absence  of  metrics  for  many  projects.  The  only  schedule-related  metric  found  to  be  used  was 
whether  the  project  met  its  planned  schedule,  and  how  much  slip  had  occurred.  And  the  limited 
schedule  data  that  are  available  are  used  on  a  project-by-project  basis  only.  No  attempt  at  any 
level  was  found  to  look  at  schedule  performance  across  a  portfolio  of  projects,  including  among 
different  development  centers,  mission  areas,  or  defense  contractors.  Thus,  no  one  knows 
whether  one  organization  is  doing  better  or  worse  in  terms  of  schedules,  or  whether  one  approach 
is  more  effective  than  another.  Without  metrics,  it  is  also  impossible  to  determine  the  schedule- 
related  performance  of  the  development  system  as  a  whole. 
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Observation  3D:  No  effort  is  made  to  compare  a  project’s  development  time  with  the  time 
that  should  be  required  to  develop  it 

Tied  to  the  issue  of  schedule-related  incentives  is  the  difference  between  planned 
development  time  and  required  development  time.  No  effort  was  found  to  identify  the  time 
necessary  to  develop  a  project,  despite  estimates  by  program  managers’  that  projects  could  be 
completed  in  half  the  remaining  time,  and  by  program  element  monitors  that  they  could  be 
completed  in  60  percent  of  the  original  time.  No  follow-up  reports  address  the  schedule. 

Because  of  the  lack  of  tools  and  information,  any  measure  of  the  required  time  is  difficult 
to  establish  with  confidence.  Nor  is  it  possible  to  determine  which  projects  are  attempting  to 
adhere  to  an  aggressive  schedule  and  which  are  simply  fulfilling  an  excessively  long  schedule. 
Schedules  lacking  an  estimate  of  required  development  time  would  appear  to  encourage  project 
managers  to  push  for  longer-than-necessary  timetables  to  ensure  that  they  can  be  met. 


Observation  3E:  The  absence  of  incentives  for  schedule  reduction  is  partly  due  to  a  lack  of 
awareness  of  the  importance  of  cycle  time  and  its  effects  on  the  overall  development 
process.  Participants  have  little  knowledge  of  commercial  practices  that  reduce 
development  time. 

The  absence  of  internal  incentives  appear  to  stem  from  a  lack  of  awareness  of  the  effects 
of  long  cycle  times,  and  of  what  can  be  done  to  shorten  them.  Most  people  involved  in  the 
development  process  want  to  do  what  is  best  for  the  country,  based  on  their  understanding  of 
priorities  and  options.  But  because  leadership  does  not  emphasize  reducing  project  schedules, 
other  objectives  receive  more  effort.  This  problem  also  stems  from  lack  of  awareness,  education, 
and  training  on  improvements  in  commercial  development  practices.  Few  people  in  the  defense 
acquisition  system  can  claim  recent  experience  in  commercial  product  development.  Military 
officers,  who  make  up  a  majority  of  the  project  managers,  enter  service  at  a  young  age  and 
cannot  exit  and  reenter.  The  few  programs  that  expose  acquisition  officers  to  industry  activities 
focus  primarily  on  defense  companies,  and  these  hands-on  programs  are  limited  and  currently 
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being  scaled  back. 

According  to  a  senior  faculty  member  at  Defense  System  Management  College,  courses 
for  project  managers  teach  approved  defense  policies  and  practices  and  not  necessarily  the  best 
available  practices.  A  review  of  those  courses  found  few,  if  any,  that  relied  on  commercial 
examples  or  experiences. 

What’s  more,  few  project  managers  studied  product  development  as  part  of  their  formal 
education.  The  surveys  showed  that  only  1  in  5  project  reported  taking  a  course  related  to 
product  development  at  a  college  or  university.  Yet,  the  recent  commercial  emphasis  on 
reducing  product  development  time  has  spurred  the  creation  of  many  university  and  continuing 
education  programs.  These  are  located  primarily  at  the  premier  schools,  which  are  not  typically 
available  to  military  officers  or  government  employees  owing  to  the  government’s  unwillingness 

to  pay  the  high  cost  of  tuition.  Very  few  officers  are  allowed  to  attend  business  schools  full 

..  110 
time. 

Without  a  common  understanding  of  commercial  development  practices,  managers  of 
defense  projects  are  unlikely  to  obtain  similar  results. 

4.  Overriding  Influence  of  Funding-Related  Constraints 

As  the  research  shows,  the  funding  process  is  the  dominate  influence  on  development, 
and  on  the  creation  and  execution  of  project  schedules.  Funding  limitations  were  found  to  be  the 
overriding  cause  of  long  cycle  times.  Funding  issues  also  consume  most  of  the  attention  that 
managers  should  be  devoting  to  supervising  the  projects  effectively. 

The  constant  focus  on  the  budget  process  changes  the  organization’s  strategic  goal  from 
quickly  satisfying  the  user’s  needs  to  fighting  for  money.  In  this  environment,  dilemmas 
associated  with  product  development  time  appear  to  be  overwhelmed  by  constant  acrimonious 
budget  battles  and  negotiations.  This  leads  to  several  funding-related  observations. 


1,0  In  1998,  the  Air  Force  has  6  officers  attending  full  time  business  schools  of  which  only  1  is  attending 
a  top  ten  rated  school. 
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Observation  4A:  Projects  are  not  funded  based  on  the  optimum  required  development 
schedule. 

Budget  requests  from  the  Program  Offices  are  based  on  the  level  of  the  funds  the  offices 
think  they  can  get,  not  on  what  they  consider  optimal  for  their  projects.  Funding  considerations 
are  driving  project  planning  to  a  far  greater  extent  than  project  planning  is  driving  project 
funding.  The  program  managers  and  program  element  monitors  estimated  that  with  full  funding 
and  a  strong  emphasis  on  short  development  time,  a  project  could  be  completed  in  half  the 
scheduled  time. 


Observation  4B:  There  are  too  many  projects  in  the  development  process  to  be  efficiently 
supported  by  the  available  resources.  This  is  a  key  cause  of  lengthening  development 
schedules. 

Across  all  types  of  projects  from  large  to  small,  from  and  revolutionary  to  incremental, 
funding  limitations  were  reported  to  be  the  primary  factor  in  determining  schedules.  Funding 
limitations  were  also  the  most  significant  barrier  to  shortening  project  schedules.  These  results 
lead  to  the  observation  that  the  resources  available  are  not  adequate  to  support  the  projects  in 
development.  Simply  put,  too  many  projects  are  competing  for  too  limited  resources.  The  result 
is  that  few  projects  are  being  completed  in  an  efficient  and  timely  manner.  (Similar  observations 
are  common  among  companies  with  significant  problems  in  their  development  systems.111) 

This  situation  is  due  to  several  factors.  One  is  that  despite  a  dramatic  decrease  in  the 
development  budget  since  1986,  the  number  of  projects  has  not  been  concomitantly  reduced. 

This  has  produced  a  virtual  logjam  in  the  development  process,  lengthening  average 
development  time  and  allowing  schedules  to  slip  owing  to  funding  limitations.  The  result  is  an 
increase  in  the  number  of  projects  in  development  at  any  one  time. 

This  is  not  a  novel  observation  for  military  development  projects,  and  it  is  not  only  found 


1,1  Clark  and  Wheelwright,  Smith  and  Reinersten,  others. 
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during  reductions  in  defense  spending.  In  1983,  during  the  Reagan  era  growth  in  defense 
spending,  the  Affordable  Acquisition  Approach  Study  found  too  many  projects  in  the 
development  pipeline  to  be  efficiently  supported.112  The  report  stated  that  programs  were  being 
squeezed  and  stretched  to  accommodate  available  funding. 

Nor  is  overextending  development  resources  unique  to  military  projects.  Most 
companies  with  significant  problems  in  their  development  system  face  a  similar  cause. .  The 
results  are  often  similar  to  the  military  experience  as  well:  long  schedules,  significant  project 
slip,  inefficient  allocation  of  resources,  and  products  that  do  not  meet  changing  customer  needs. 
In  the  commercial  marketplace,  however,  competition  provides  effective  feedback,  as  other 
companies  with  more  effective  development  processes  boost  their  market  share  and  profits,  and 
inefficient  firms  either  improve  or  go  out  of  business. 


Observation  4C:  There  are  few  limits  on  the  number  of  projects  in  the  development 
process.  Any  limits  that  do  exist  are  primarily  based  on  the  ability  of  a  Program  Office  to 
obtain  funding. 

Another  problem  is  an  inability  or  unwillingness  on  the  part  of  Program  Offices  to  limit 
the  number  of  projects  entering  or  remaining  in  the  development  process.  The  primary 
consideration  in  such  decisions  is  whether  development  plans  are  adequate  and  the  required 
funds  can  be  located.  As  the  research  showed,  project  schedules  are  most  often  changed  to 
reflect  the  funding.  Because  no  apparent  formal  limits  exist,  significant  effort  failed  to  identify 
the  total  number  of  military  products  in  development. 


”2  The  Affordable  Acquisition  Approach  Study.  Air  Force  Systems  Command  1983. 
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Observation  4D:  There  is  no  effective  screening  process  to  control  the  number  and  types  of 
projects  entering  development  and  to  ensure  that  they  are  optimally  planned 

The  inability  to  limit  the  number  of  projects  results  partly  from  the  lack  of  an  effective 
screening  and  selection  process.  Despite  the  fact  that  nearly  all  resources  for  each  mission  area 
are  supposed  to  be  allocated  through  the  modernization  planning  process,  42  percent  of  all 
projects  surveyed  were  initiated  outside  that  process  by  senior  leadership.  It  could  not  be 
determined  if  other  projects  were  canceled  or  their  funding  levels  were  reduced  to  make  room  in 
the  budget  for  these  added  projects.  The  reasons  so  many  are  initiated  outside  the  formal  process 
may  include  changing  requirements  that  biannual  planning  does  not  account  for,  a  lack  of  faith  in 
the  results,  or  a  lack  of  discipline  in  adhering  to  the  results.  One  very  senior  former  Air  Force 
official  expressed  displeasure  at  how  the  modernization  planning  process  worked  and  its  inability 
to  stay  within  expected  funding  allocations,  and  indicated  that  the  process  was  turning  into  a 
formality  rather  than  a  serious  exercise. 

Decisions  as  to  which  projects  were  begun,  continued,  or  advanced  appeared  to  be  spread 
among  various  major  commands  and  Pentagon  organizations  based  on  the  negotiated  funding 
decisions.  The  decision  to  advance  a  project  to  the  next  stage  did  not  appear  to  be  based  directly 
on  the  need  for  the  system  or  the  performance  of  the  project.  The  process  of  generating  and 
validating  requirements  did  not  appear  to  effectively  limit  the  number  of  projects.  Hundreds  of 
approved  operational  requirements  documents  and  mission  needs  statements  exist  for  unfunded 
projects.  In  many  cases  when  senior  leadership  initiates  a  project,  the  project  begins  before  the 
requirements  document  is  complete.  The  requirements  processes  were  found  to  slow  the 
initiation  of  some  projects  in  specific  instances  where  key  leaders  doubted  the  importance  of  the 
project,  or  the  project  threatened  other  budget-related  priorities.  Many  of  the  arguments  that 
occur  within  the  requirements  processes  appear  to  be  due  to  their  impact  on  resource  allocations, 
and  do  not  concern  operational  requirements. 

The  acquisition  milestone  decisions  as  now  used  do  not  limit  the  number  of  projects 
initiated  or  in  development.  The  milestone  decisions  primarily  ensure  that  a  project  has  met  the 
requirements  needed  to  enter  the  next  phase,  and  that  the  funds  to  support  the  project  as  planned 
are  allocated  within  the  program  objective  memorandum  (POM).  The  milestone  decision 
process  is  not  used  to  determine  which  projects,  which  type  of  projects,  or  how  many  projects 
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should  be  in  the  development  process  at  any  one  time.  A  project  almost  always  passes  a 
milestone  decision  unless  it  has  obvious  technical  problems  or  a  significant  shortfall  between 
planned  and  allocated  resources.  If  a  significant  funding  shortfall  exists,  the  project  is  usually 
rephased  to  ensure  that  the  funds  and  the  plan  match. 

For  their  part,  the  major  commands  establish  a  preferred  order  for  projects  but  do  not 
determine  the  number  initiated.  Each  major  command  tries  to  begin  as  many  of  its  desired 
projects  as  possible  based  on  the  results  of  the  funding  process. 

The  resource  allocation  process  is  the  primary  method  used  to  limit  entry  to  the 
development  process,  but  even  this  process  does  not  effectively  control  the  number  of  projects  . 
The  Pentagon  resource  allocation  process  is  a  multi-level,  broad-based,  staff-driven  process 
through  which  participants  attempt  to  gain  the  “corporate  position.”  Any  such  process  is  more 
likely  to  yield  compromises  and  incremental  changes  rather  than  bold  decisions,  allocating  less 
funding  than  requested  to  competing  projects.  The  use  of  the  resource  process  to  select  new 
projects  and  control  entry  is  undermined  by  the  practice  of  underestimating  a  project’s  initial 
cost,  and  the  use  of  optimistic  projected  funding  levels.  Other  than  the  elaborate  funding 
processes,  no  other  central  organization  or  process  controls  the  number  of  projects  initiated. 

The  fact  that  the  funding  process,  or  any  other  process,  does  not  limit  the  number  of 
projects  to  remain  within  the  number  of  projects  that  can  be  most  effectively  executed  is  evident 
by  the  number  of  projects  that  are  funding  limited  at  initiation,  and  the  number  that  later 
experience  slip  owing  to  funding  instability. 


Observation  4E:  There  are  no  funds  available  to  accelerate  projects,  and  the  lack  of  such 
funds  is  a  major  impediment  shortening  project  development  schedules. 

The  fact  that  thousands  of  desired  projects  are  waiting  to  be  initiated  means  that  100 
percent  of  development  funds  are  committed  to  various  projects  at  any  one  time.  Funds  that 
Program  Offices  do  not  committed  to  specific  projects  may  be  lost  to  other  offices,  the  major 
commands,  and  other  services.  Cost  overruns  and  other  must-pay  bills  absorb  any  additional 
funds  that  might  have  been  used  to  accelerate  individual  projects.  The  result  is  that  only  a  small 
percentage  of  projects  come  in  ahead  of  schedule. 


330  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


Summary  of  Observations 

These  observations,  while  painting  a  stark  picture  of  the  defense  development  process, 
actually  provide  evidence  that  the  system  can  significantly  improve.  Areas  in  need  of  attention 
include  schedule-related  information  and  tools,  incentives,  and  the  processes  for  selecting 
projects  and  allocating  funds.  Few  of  these  sectors  require  changes  in  law  or  higher-level 
regulations,  but  most  do  entail  changes  in  organizational  emphasis  and  leadership  direction. 
Implementing  the  changes  will  require  strong  leadership  and  hard  decisions. 

The  people  interviewed  are  hard-working,  well-intentioned,  and  doing  what  they  believe 
is  in  the  best  interest  of  the  country  and  their  military  service.  Those  people  are  not  the  problem. 
Rather,  the  process  drives  them  to  actions  that  seem  optimal  from  a  project  standpoint  but  that 
produce  less-than-optimal  results  for  the  system  as  a  whole. 
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Summary  the  Observations 

1.  Lack  of  Leadership  and  Emphasis  on  Cycle  Time 

Observation  1A:  There  has  been  little  leadership  on  the  issue  of  reducing  product  development  cycle 
time.  Reducing  Product  Development  Schedules  is  not  currently  seen  as  an  important  priority  of 
the  product  development  system. 

Observation  IB:  There  is  currently  no  visible  or  widely  held  goal  or  objective  for  cycle  time  reduction 

Observation  1C:  There  is  currently  no  strong  advocate  for  cycle  time  reduction 

2.  Schedule  Information  and  Tools  in  the  Development  Process 

Observations  2A:  Schedules  are  currently  based  on  factors  other  than  the  actual  time  required  to 
develop  the  project.  Once  established  these  schedules  determine  the  minimum  time  it  will  take  to 
develop  the  project. 

Observation  2B:  Little  rigorous  schedule  based  information  is  available  to  be  used  to  plan,  manage, 
or  evaluate  the  schedules  of  development  projects. 

Observation  2C:  There  are  no  effective,  accepted,  or  widely  used  schedule  estimating  tools  for 
military  development  projects.  The  lack  of  effective  schedule  estimating  tools  limits  the  ability  to 
make  decision  based  on  the  schedule  information  available. 

Observation  2D:  Contractors  base  their  proposed  schedules  on  the  government’s  expected  schedule 
and  not  the  analysis  of  the  actual  time  required  for  the  development  phase  of  the  project. 

3.  Lack  of  Schedule  Based  Incentives 

Observation  3A:  There  are  no  effective  incentives  for  cycle  time  reduction  at  either  the  Program 
Office  or  Pentagon  levels. 

Observation  3B:  There  are  no  effective  incentives  for  cycle  time  reduction  at  the  contractor  level. 
Schedule  incentives  in  the  contracting  phase  are  only  to  meet  the  government  expected  schedule. 
There  are  few  contract  incentives  for  cycle  time  reduction  during  the  development  phase. 

Observation  3C:  There  are  no  effective  measures  or  metrics  that  are  used  to  evaluate  the  cycle  time 
performance  of  a  project.  As  such  meaningful  schedule-based  metrics  are  difficult  to  establish. 

Observation  3D:  No  use  of  any  measure  or  estimate  comparing  the  planned  or  actual  development 
time  to  the  time  necessary  to  develop  a  project  was  found. 

Observation  3E:  The  lack  of  incentive  for  schedule  reduction  is  in  part  due  to  a  lack  of  awareness  of 
the  importance  of  cycle  time  and  its  effects  on  the  overall  development  process  or  the  ability  to  do 
anything  about  it.  There  is  little  awareness  of  the  current  commercial  methods  and  practices 
available  to  reduce  development  time  or  of  their  success  in  doing  so. 

4.  Overriding  Influence  of  Funding  Related  Constraints 

Observation  4A:  Projects  are  not  funded  based  on  the  optimum  project  development  required 
schedule  but  based  on  other  funding  constraints 

Observation  4B:  There  are  currently  too  many  projects  in  the  development  process  to  be  efficiently 
supported  by  the  available  resources.  This  is  a  key  cause  of  the  lengthening  development  schedules. 

Observation  4C:  There  are  no  current  limits  on  the  number  of  projects  in  the  development  process.  Any 
current  limits  that  do  exist  are  primarily  based  on  the  ability  of  the  project  office  to  obtain  funding. 

Observation  4D:  There  is  not  an  effective  project  screening  process  to  control  the  number  and  types 
of  projects  entering  the  development  process  and  to  ensure  that  they  are  optimally  planned. 

Observation  4E:  There  are  no  funds  available  to  accelerate  projects;  and  the  lack  of  such  funds  is  a 
major  impediment  to  efforts  to  reduce  project  development  schedules. 


Table  13-18:  Summary  of  Observations  of  the  DoD  Product  Development  Process 
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Policy  Recommendations  and  Development 
Strategies _ 


The  observations  on  the  overall  development  process  lead  to  specific  recommendations  for 
shortening  cycle  times  for  military  products.  These  recommendations  are  not  a  complete  set  of  the 
needed  actions  but  include  the  necessary  first  steps  toward  making  the  process  faster  and  more  effective. 
A  shorter  development  process  will  allow  for  a  change  in  the  strategy  used  to  supply  warfighters  with 
new  and  modified  systems — one  that  provides  military  capability  when  needed  and  allows  the  armed 
forces  to  quickly  adapt  to  new  technologies  and  a  changing  world.  Both  the  specific  policy 
recommendations  and  the  changes  in  overall  development  strategy  they  make  possible  are  discussed  in 
this  chapter. 
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A.  Recommendations  for  Shortening  Product  Development  Time 

This  research  leads  to  five  sets  of  recommendations  that  will  reduce  the  programmatic 
constraints  on  military  product  development  time.  These  recommendations  address  what  was 
found  to  be  the  limiting  factor  in  shortening  cycle  times.  They  encompass  focusing  encompass 
leadership,  information,  incentives,  resources,  and  demonstrated  results. 


Overarching  recommendations: 

1.  Provide  clear  leadership  on  reducing  cycle  time. 

2.  Develop  and  use  schedule-based  information. 

3.  Provide  incentives  for  reducing  cycle  time. 

4.  Mitigate  funding-based  constraints  on  development  projects. 

5.  Show  results  through  near-term  demonstration  projects. 


Within  each  of  these  areas  are  specific  steps  to  remove  programmatic  constraints.  The 
recommendations  do  not  address  the  organizational  and  managerial  changes  contractors  and 
Program  Offices  need  to  make,  as  described  in  Chapter  4.  However,  those  factors  do  not  appear 
to  be  the  primary  barrier  to  shorter  development  cycles.  But  as  programmatic  limitations  decline, 
those  factors  will  have  to  be  addressed  to  further  reduce  development  time.  Until  then,  much 
needs  to  be  done  in  areas  controlled  by  government. 

The  most  important  recommendation  is  to  adopt  a  time-driven  development  strategy,  and 
to  make  short  cycle  time  an  important  priority  for  each  project.  Today’s  10-to-  12-year  average 
development  time  cannot  keep  pace  with  technological  change  or  respond  to,  emerging  threats, 
and  it  raises  costs  significantly. 

The  Packard  Commission  recognized  the  importance  of  cycle  time  when  it  stated  that 
excessively  long  development  times  were  the  central  problem  in  defense  acquisition.  The 
commission’s  core  recommendation  was  to  cut  these  times  in  half  for  major  weapon  systems. 

But  although  the  commission’s  recommendations  for  reaching  this  goal  were  implemented,  cycle 
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time  did  not  shrink:  the  acquisition  system  did  not  change  from  one  focused  on  budget  and 
performance  to  one  concerned  with  quickly  meeting  users’  needs. 

Without  this  overriding  objective,  it  is  this  author’s  view  that  the  system  will  not  change 
and  cannot  change.  If  cycle  times  do  not  become  a  significant  focus  they  will  likely  continue  to 
grow  and  the  defense  development  process  will  remain  ineffective  and  inefficient. 

Making  the  decision  to  change  the  major  focus  of  development  is  not  easy,  nor  will  such 
a  change  likely  happen  quickly.  Significant  obstacles  must  be  overcome.  Below  are  specific 
recommendations  to  begin  the  process  of  overcoming  those  barriers.  (For  additional  information 
on  the  research  supporting  these  recommendations  and  their  implications,  see  Appendix  1.) 

Recommendation  1:  Provide  clear  leadership  on  reducing  development  cycle 
time. 

Reorienting  organizations  to  accept  schedule  as  an  important  factor  and  shorter  cycle 
times  as  an  important  objective,  will  require  significant  leadership  at  all  levels.  Leaders  will 
have  to  show  that  shortening  development  time  relates  directly  to  the  ability  to  provide  an 
effective  and  affordable  defense.  Leaders  will  also  have  to  set  aggressive  goals  for  reducing 
cycle  times  and  remain  strong  and  powerful  advocates  for  them,  holding  organizations 
accountable  for  meeting  the  goals. 

1.1:  Make  a  clear  business  case  for  reducing  development  cycle  time. 

1.2:  Establish  quantifiable  goals  for  reducing  development  cycle  times. 

1.3:  Appoint  an  advocate  for  reducing  development  cycle  time. 


Recommendation 

Recommendation 

Recommendation 
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Recommendation  2:  Develop  and  use  rigorous  schedule-based  information 
and  tools. 

Accurate  information  and  analysis  are  required  to  make  informed  decisions  on  project 
schedules.  Today  schedules  are  primarily  based  on  expected  funding  levels  and  not  on  rigorous 
evaluation  partly  because  of  a  lack  available  and  easy-to-use  tools.  Three  steps  could  help 
address  that  need.  First,  a  formal  estimate  of  development  time,  and  an  analysis  of  the  factors 
limiting  schedule,  must  be  part  of  the  project  approval  process.  This  could  take  the  form  of  a 
“should  take”  analysis--  similar  to  the  “should  cost”  estimate  of  how  much  a  project  would  cost 
if  done  as  efficiently  as  possible.  Such  an  analysis  would  rely  on  a  comparison  with  other 
projects,  parametric  estimation  of  the  required  time,  and  a  bottom-up  estimate  stemming  from 
engineering  requirements.  This  analysis  would  provide  senior  leaders  with  the  information  they 
need  to  make  informed  scheduling  decisions  and  allow  them  to  establish  realistic  project  goals. 
Such  an  analysis  would  also  highlight  the  barriers  to  shortening  project  schedules  early  in  a 
project’s  planning  phase.  Similar  analysis  should  be  included  by  the  contractors  when  presenting 
their  proposed  schedules. 

Recommendation  2.1:  Base  initial  project  plans  on  development-related  requirements  and 
perform  a  “should  take”  analysis. 

Recommendation  2.2:  Develop  and  use  effective  tools  for  estimating  and  evaluating  project 
schedules. 

Recommendation  2.3:  Require  contractors  to  include  a  “schedule  proposal”  section  in  their 
overall  proposals. 


Recommendation  3:  Provide  incentives  that  encourage  shorter  cycle  times. 

Some  negative  schedule-related  incentives  may  change  when  leaders  place  a  greater 
emphasis  on  shortening  project  schedules  and  explain  their  rationale.  But  even  with  such 
leadership,  strong  disincentives  to  reducing  cycle  time  will  have  to  be  consciously  removed  and 
significant  positive  incentives  established  for  the  Pentagon,  Program  Office  and  defense 
contractors  for  significant  changes  can  occur. 
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Incentives  for  contractors  can  be  applied  both  before  and  after  the  contract  award. 

Making  the  length  and  risk  of  the  proposed  schedule  a  significant  criterion  for  winning  a  contract 
will  encourage  companies  to  schedules  that  are  both  short  and  achievable.  Then,  incentives  in 
the  contract  itself  can  encourage  contractors  to  achieve  the  planned  schedule  and  further  shorten 
it.  Contracts  should  also  include  penalties  for  as  well  failure  to  achieve  the  specified  schedule. 

A  significant  training  program  must  be  established  to  convince  the  defense  acquisition 
community,  including  contractors,  that  shortening  cycle  time  is  important,  that  it  is  possible,  and 
that  the  commercial  sector  has  done  so  through  a  variety  of  means  that  can  be  emulated.  Such  an 
effort  will  require  an  awareness  of  the  best  product  development  practices  from  a  wide  array  of 
industries. 

Recommendation  3.1:  Provide  incentives  at  the  Program  Office  and  Pentagon  levels. 
Recommendation  3.2:  Provide  incentives  at  the  contractor  level. 

Recommendation  3.2.1:  Make  the  length  of  the  development  schedule  and  the  associated 

risk  a  significant  criterion  in  contractor  selection. 

Recommendation  3.2.2:  Provide  significant  schedule-based  contract  incentives. 

Recommendation  3.3:  Provide  extensive  training  on  best  product  development  practices  from 
commercial  industry. 


Recommendation  4:  Mitigate  funding-based  schedule  limitations. 

The  foregoing  recommendations  are  unlikely  to  have  a  noticeable  impact  without 
significant  changes  in  the  method  by  which  the  services  and  DoD  fund  development  projects.  To 
create  products  efficiently  and  effectively,  projects  must  be  funded  at  the  levels  determined  by 
their  development-related  requirements.  And  each  should  be  funded  according  to  its  most 
effective  and  efficient  schedule.  These  changes  will  require  controlling  the  number  of  projects 
under  way  at  any  one  time. 

All  these  goals  can  be  achieved  by  establishing  a  formal  screening  process  to  determine 
which  projects  are  highest  priority  and  which  Can  be  fully  funded  within  the  available  resources 
and  the  planned  time.  A  multiple-level  screening  process  would  also  allow  projects  to  compete 
for  approval  to  proceed  to  the  next  development  phase. 
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To  ensure  sufficient  funding  to  fully  support  each  project,  the  number  of  projects  within 
each  phase  at  any  given  time  should  be  limited.  This  could  limit  could  be  reached  by  completing 
many  projects  more  quickly  while  delaying,  postponing,  or  canceling  others. . 

Once  a  project  is  in  the  development  process,  it  is  important  to  ensure  that  it  can  move 
through  it  as  quickly  as  possible.  To  make  that  possible,  funds  for  accelerating  the  schedule  as 
opportunities  arise  must  be  readily  available.  Priority  in  the  use  of  funds  that  become  must  be 
given  to  accelerating  and  completing  existing  projects  rather  than  to  initiating  new  ones. 


Recommendation  4.1:  Require  that  all  projects  initiated  be  fully  funded  based  on  development- 
related  requirements. 

Recommendation  4.2:  Establish  an  effective  project  screening  process. 

Recommendation  4.3:  Limit  the  number  of  projects  in  each  phase  of  development. 
Recommendation  4.4:  Clear  the  logjam  of  current  projects. 

Recommendation  4.5:  Ensure  that  resources  are  available  to  accelerate  projects  as  opportunities 
arise. 
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Recommendation  5:  Show  results  through  near-term  demonstration  projects. 

Since  is  DoD  has  little  experience  in  reducing  cycle  times,  and  lessons  from  the 
commercial  sector  are  not  always  directly  applicable,  demonstrations  can  help  verify  the  impact 
of  the  aforementioned  changes. 

The  effect  of  many  recommendations,  including  enhancing  the  incentives  for  reducing 
development  time,  can  be  observed  by  selecting  individual  projects  as  demonstration  vehicles. 
The  impact  of  other  recommendations,  such  as  establishing  a  screening  process,  can  be  seen  only 
when  applied  to  a  portfolio  of  projects.  In  that  case,  a  Program  Office  that  oversees  a  large 
number  of  smaller  projects  can  be  selected  as  a  “reinvention  laboratory”  and  given  wide  leeway 
to  implement  the  recommended  changes.  While  this  would  require  significant  coordination  and 
high-level  support,  it  could  demonstrate  the  cumulative  and  interactive  effects  of  the 
recommendations  across  an  entire  development  organization.  Such  a  result  would  give  DoD 
confidence  to  adopt  the  recommendations  across  all  weapon  systems. 

Once  the  ability  to  produce  products  quickly,  effectively,  and  reliably  is  demonstrated, 
significant  other  changes  can  occur  in  the  strategy  use  to  develop  weapons. 


Recommendation  5.1:  Establish  pathfinder  projects  that  demonstrate  the  effectiveness  of  the 
cycle  time  reduction  approach. 


Recommendation  5.2:  Establish  a  reinvention  laboratory  that  demonstrates  the  effectiveness  of 
the  approach  on  products  within  an  entire  Program  Office  or  Product 
Center. 
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Summary  Table  of  Recommendations 

Recommendation  1:  Provide  Clear  Leadership  on  Reducing  Cycle  Time 

1.1  Make  a  Clear  Business  Case  for  Reducing  Cycle  Time 

1.2  Establish  Quantifiable  Goals  for  Reducing  Cycle  Time 

1.3  Appoint  an  Advocate  for  Reducing  Cycle  Time 

Recommendation  2:  Develop  and  Use  Rigorous  Schedule-Based  Information  and  Tools 

2.1  Base  Initial  Project  Plans  on  Development-Related  Requirements  and  Perform  a  “Should  Take” 
Analysis 

2.2  Develop  and  Use  Effective  Tools  Estimating  and  Evaluating  Project  Schedules 

2.3  Require  a  “Schedule  Proposal”  Section  in  Contractor  Proposals 

Recommendation  3:  Provide  Incentives  That  Encourage  Cuts  in  Cycle  Time 

3.1  Provide  Incentives  at  the  Program  Office  and  Pentagon  Levels 

3.2  Provide  Incentives  at  the  Contractor  Level 

3.2.1  Make  the  Length  of  the  Development  Schedule  and  Associated  Risk  Significant  Criteria  in 

Contractor  Selection 

3.2.2  Provide  Significant  Schedule-Based  Contract  Incentives 

3.3  Provide  Extensive  Training  on  Best  Development  Practices 

Recommendation  4:  Mitigate  Funding-Based  Schedule  Limitations 

4.1  Require  All  Projects  to  Be  Fully  Funded  Based  on  Development-Related  Requirements 

4.2  Establish  an  Effective  Project  Screening  Process 

4.3  Limit  the  Number  of  Projects  in  Each  Phase  of  Development 

4.4  Clear  the  Logjam  of  Current  Projects 

4.5  Ensure  That  Resources  Are  Available  to  Accelerate  Projects  as  Opportunities  Arise 

Recommendation  5:  Show  Results  through  Near-Term  Demonstration  Projects 

5.1  Establish  Pathfinder  Projects  That  Demonstrate  the  Effectiveness  of  the  Approach  to  Reducing  Cycle 
Time 

5.2  Establish  Reinvention  Laboratories  That  Demonstrates  the  Results  within  an  Entire  Program  Office 
or  Product  Center 


Table  13-19:  Recommended  Actions  to  Reduce  Development  Time  for  Defense  Products 
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B.  Longer-Term  Recommendations  for  Changing  the  Development  Strategy 

Shortening  the  time  required  to  develop  weapon  systems  is  important  in  developing  better 
systems  faster  and  at  lower  cost.  However,  it  also  represents  a  key  step  in  allowing  a  larger 
change  in  the  way  military  systems  are  developed  and  fielded.  A  fast  development  process  will 
provide  the  U.S.  with  a  long-term,  sustainable  military  capability  to  respond  quickly  to  changing 
threats  and  emerging  technologies.  A  shift  in  strategy  from  developing  a  wide  array  of  systems 
to  cover  all  possible  emerging  threats  to  one  that  is  more  responsive  to  specific  threats  will 
ensure  that  the  right  weapons  are  developed  and  ready  to  be  fielded  at  the  right  time. 

i 

The  military  worth  of  a  system  is  based  on  two  inherent  aspects.  The  first  is  its 
performance  advantage  over  an  opposing  system.  The  second,  often  not  fully  considered,  is  the 
length  of  time  that  system  can  retain  its  technical  advantage  and  avoid  compromise  by 
countermeasures.  To  head  off  such  compromise,  the  services  usually  call  for  significant 
technological  superiority  in  a  broad  range  of  areas  to  ensure  dominance  even  if  an  enemy  finds  a 
way  to  counter  a  specific  system.  Thus,  the  U.S.  Air  Force  now  builds  systems  “just-in-case”~as 
insurance  against  a  “pop-up”  threat.  This  is  one  argument  for  the  F-22  and  Joint  Strike  Fighter, 
both  intended  to  counter  threats  that  may  develop  over  the  next  30  years. 

The  net  effect  is  that  many  systems  are  developed  to  meet  threats  that  never  materialize, 
and  when  systems  are  finally  fielded  they  face  a  dramatically  different  mission  than  they  were 
designed  for.  While  Pentagon  respondents  indicated  that  70  percent  of  projects  were  designed  to 
meet  current  operational  deficiencies,  they  indicated  that  fully  30  percent  were  intended  to  meet 
projected  needs  that  may  not  materialize.  This  leads  to  the  worst  type  of  inefficiency — 
developing  and  producing  the  wrong  weapon  at  the  wrong  time  with  obsolete  technology.  The 
B-2  and  Milstar  systems  are  visible  examples  of  systems  designed  to  meet  a  mission  that  never 
materialized  or  disappeared  before  they  were  completed.  Such  systems  are  then  pushed  into  a 
service  for  which  they  were  not  designed  and  may  be  ill  suited.  A  specific  example  is  using  the 
Milstar  as  a  tactical  communications  terminal  instead  of  as  a  strategic  system  during  the  second 
stage  of  a  nuclear  exchange. 
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The  just-in-case  strategy  forces  the  U.S.  to  spread  its  resources  over  many  systems,  a 
practice  that  results  in  longer  development  times.  Revamping  the  basic  strategy  from  just-in-case 
to  just-in-time  requires  two  changes.  First,  the  U.S.  must  be  able  to  develop  new  systems 
quickly,  and  second,  it  must  be  able  to  produce  enough  units  rapidly. 

T  he  U.S.  now  takes  10  to  20  years  to  develop  and  field  major  weapon  systems.  These 
systems  are  expected  to  be  in  operation  for  20  to  25  years.  That  forces  the  country  to  project 
threats  and  military  and  commercial  technologies  40  years  into  the  future — an  impossible  task. 
Such  an  unrealistic  goal  is  what  drives  the  demanding  performance  requirements— and  the 
resulting  high  costs-off  military  systems. 

The  recent  response  o  long  cycle  times  has  been  to  establish  a  long-term  planning 
organization  in  the  Pentagon  to  project  required  force  structure,  weapons,  and  technologies  25  to 
40  years  in  advance.  The  effect  of  these  plans  is  to  lock  in  systems  and  related  schedules  long 
before  it  is  possible  to  know  the  actual  threats.  Once  made  these  plans  and  schedules  prove  very 
difficult  and  costly  to  change. 

The  system  is  most  closely  comparable  to  the  central  planning  function  of  Communist 
countries  with  few  market  or  entrepreneurial  forces  at  work.  Bureaucratic  processes  dominate 
planning  and  the  results  are  long  development  schedules  and  too  many  projects  in  development. 
The  system  also  forces  developers  to  make  large  technological  leaps  to  ensure  that  systems  are 
technically  superior  when  they  are  eventually  fielded — further  exacerbating  the  problems. . 

A  just-in-time  strategy  would  allow  a  more  targeted  approach  in  deciding  which  systems 
are  needed  to  counter  real  threats.  It  would  also  reduce  U.S.  vulnerability  to  unpredicted  enemy 
systems  based  on  new  technologies  or  novel  combinations  of  existing  technologies. 

A  just-in-time  approach  would  not  be  new-it  was  the  traditional  U.S.  military  strategy 
from  the  Revolution  through  the  Second  World  War.  Only  during  the  Cold  War  did  the  U.S.  feel 
the  need  to  maintain  long-term  technological  superiority  over  all  enemies.  During  World  War  II, 
the  U.S.  developed  better  systems  quickly  and  produced  them  massively.  But  the  speed  and 
potential  effects  of  nuclear  war  and  our  unpreparedness  for  the  Korean  War  made  the  U.S.  fear 
being  caught  unprepared.  This  developed  into  the  idea  of  the  “come  as  you  are  war,”  which 
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would  be  fought  in  Europe,  with  the  ultimate  winner  determined  in  a  matter  of  weeks  before  any 
production  could  affect  the  outcome.  I  believe  the  chance  of  this  type  of  total  lightning  war  has 
been  thoroughly  eliminated  by  the  changes  in  Russia  and  the  lessening  of  the  perceived  nuclear 
threat.  Smaller  operations  will  still  be  on  a  “come  as  you  are”  basis  but  will  require  only 
significant— as  opposed  to  massive— inventories  of  the  highest-technology  weapons. 

A  fast  development  strategy  would  rely  on  critical  technologies  and  subsystem 
components  that  are  developed  to  the  point  where  they  can  be  assembled  quickly,  tested,  and 
then  produced  as  needed  en  masse.  Such  key  technologies  would  be  in  the  areas  of  sensors, 
signal  and  computer  processors,  communication  systems,  warheads,  autonomous  control,  and 
navigation.  A  “develop  and  test  but  not  produce”  strategy  would  keep  the  U.S.  aerospace 
industry’s  product  development  capability  in  shape  to  ensure  that  it  could  be  called  upon  when 
needed.  Such  a  strategy  would  allow  for  continual  design  improvements  without  the  cost  of 
actually  implementing  the  modifications.  This  strategy  is  similar  to  the  design,  build,  and  test 
policies,  known  as  Silver  Bullet,  that  surfaced  during  the  early  1990s.  But  those  policies  were 
proposed  more  because  of  a  lack  of  production  money  rather  than  a  shift  in  overall  military 
development  strategy.  The  Silver  Bullet  strategy  was  not  widely  accepted  for  a  variety  of 
political  reasons,  which  may  have  changed  since  the  end  of  the  Cold  War  and  the  move  toward 
lower  defense  budgets. 

The  central  enabling  aspect  of  a  just-in-time  strategy  would  be  reliance  on  a  “civil  reserve 
industrial  capacity,”  much  as  the  Air  Force  relies  on  the  Civil  Reserve  Air  Fleet  to  augment  its 
transport  aircraft  with  commercial  aircraft.  This  requires  that  military  and  commercial 
production  capacities  be  merged.  Such  a  merger  is  the  major  goal  of  Dr.  Kaminski’s,  the  former 
Defense  Acquisition  Executive,  efforts  to  remove  military  specifications  and  production 
standards  in  favor  of  single  commercial  standards  and  processes."3  This  strategy  would  allow  the 
U.S.  to  maintain  defense  capacity  for  times  of  emergency  without  incurring  the  cost  of  constantly 
producing  defense  products  that  we  do  not  need  at  the  present  time.  The  strategy  also  allows  a 
move  away  from  the  need  for  constant  care  and  feeding  of  the  “defense  industrial  complex.” 


113  Statement  of  the  Undersecretary  of  Defense  for  Acquisition  and  Technology,  Paul  G.  Kaminski  before  the 
Subcommittee  on  Defense  Technology.,  Acquisition  and  Industrial  Base  of  the  Senate  Committee  on  Armed 
Services  on  Dual  Use  Technology.  May  17,  1995. 
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The  DoD  would  pay  commercial  manufacturers  to  maintain  the  ability  to  shift  rapidly  to 
production  of  military  systems. 

But  before  that  can  happen,  the  defense  industry  must  become  lean  enough  to  compete  in 
the  commercial  market,  or  commercial  firms  must  be  allowed  to  compete  in  the  defense  market 
without  large  barriers.  If  one  reads  many  of  the  Defense  Science  Board  studies  led  by  Dr. 
William  Perry  at  the  height  of  military  spending  in  the  late  1980s,  one  sees  a  merger  of  defense 
and  commercial  industry  as  a  central  theme  for  this  very  reason."4  Such  a  merger  is  starting  to 
occur  in  the  satellite,  electronics,  and  engine  industries,  but  has  not  yet  occurred  in  the  aircraft  or 
munitions  areas. 

Both  political  and  technological  reasons  make  a  change  from  just-in-case  to  just-in-time 
strategy  not  only  desirable  but  also  possible  today  as  opposed  to  10  years  ago.  Key  military 
technologies  are  now  drawn  primarily  from  the  commercial  sector,  which  moves  faster  than 
military  efforts.  Also,  open  architectures  for  commercial  computer  systems  are  being  used  in 
military  systems,  the  quality  of  commercial  products  often  exceeds  military  standards,  and  a 
policy  of  replacing  rather  than  repairing  military  components  reduces  maintenance  concerns. 

Political  changes  include  reduction  of  the  military  threat  and  a  period  of  clear  military 
superiority,  which  provides  time  to  reorganize  without  risk.  Changes  also  include  acceptance  of 
substantially  lower  long-term  defense  budgets,  the  diminished  threat  of  global  war  and  a  focus  on 
regional  conflicts,  and  the  recognition  by  industry  and  military  leaders  that  things  can  and  must 
change  if  we  are  to  provide  effective  weapons  over  a  longer  period. 

The  demonstrated  capability  to  quickly  develop  and  produce  weapons  is  the  key  factor 
enabling  such  changes  to  occur.  I  propose  that  the  long  and  difficult  process  required  to  shorten 
development  cycle  times  and  increase  the  effectiveness  of  our  development  system  begin  now, 
based  on  these  recommendations. 


"4  Use  of  Commercial  Components  in  Military  Equipment.  Final  Report  of  the  Defense  Science  Board  1986 
Summer  Study.  January  1987.  Office  of  the  Undersecretary  of  Defense  for  Acquisition. 
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Underlying  Principles 
and  Assumptions 


•  The  development  system  as  a  whole  will  be 
improved  by  optimizing  each  development  project 

•  Schedule  based  decision  must  be  based  on  accurate 
and  available  information 

•  The  contractor  can  better  determine  how  long  it  will 
take  them  to  develop  a  project  based  on  their 
capabilities  and  their  technical  approach  than  the 
government 

•  The  government  has  a  responsibility  to  ensure  funds 
are  spent  wisely 

•  The  results  of  development  efforts  can  not  be  fully 
predicted  in  advance 
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Overarching  Recommendations 


1.  Provide  Clear  Leadership  For  Cycle  Time  Reduction 

2.  Develop  and  Use  Schedule  Based  Information 

3.  Provide  Incentives  for  Cycle  Time  Reduction 

4.  Mitigate  Funding  Based  Constraints  on 
Development  Projects 

5.  Demonstrate  Application  and  Results  Through  Near 
Term  Demonstration  Projects 
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iNrriAmT^^  Recommendation  1 

Provide  Clear  Leadership 
on  Cycle  Time  Reduction 

Clear  and  decisive  leadership  at  all  levels  will  be 
required  to  make  cycle  time  reduction  a  significant 
objective  and  focus  of  the  development  process. 

Recommendations: 


1.1  Develop  The  Clear  Business  Case  for  Cycle  Time  Reduction 

1.2  Establish  Clear  Quantifiable  Cycle  Time  Reduction  Goal 

1.3  Appoint  An  Advocate  or  Champion  for  Cycle  Time  Reduction 
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1.  Provide  Clear  Leadership 
on  Cycle  Time  Reduction 


Research  Findings: 

-  No  consensus  of  the  importance  or  effects  of  long  cycle 
times  or  reduction  of  cycle  times 

-  Fourth  of  4  in  priority  at  all  levels 

-  No  accepted  goals  of  cycle  time  reduction 

-  Current  goal  is  to  meet  schedules  not  reduce  them 

-  No  organization  assigned  responsibility  for  cycle  times 

-  FM  budget  focused 

-  Users  performance/budget  focused 

-  AQ  budget/performance/technology/new  system  focused 
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Recommendation  1.1:  Develop  the  Clear 
Business  Case  for  Product 
Development  Cycle  Time  Reduction 


Research  Findings: 

-  There  is  no  clear  or  widely  accepted  understanding  of  importance  or  impact  of 
shortening  development  cycle  time 

-  Widely  disparate  and  contradictory  views  encountered  across  all  organizations 
and  all  levels  on  the  importance  and  benefits  of  cycle  time  reduction 

-  Shorting  project  schedule  is  currently  seen  as  the  last  priority 

Implications: 

-  Must  develop  an  authoritative  and  widely  accepted  case  for  cycle  time 
reduction  and  the  impact  of  the  current  long  cycle  times 

-  Identify  and  estimate  all  impacts,  both  the  positive  and  the  negative,  of 
reducing  cycle  time  at  the  individual  project  and  portfolio  of  projects  levels 

-  Should  directly  address  the  effect  on  total  and  annual  cost;  requirements, 
technology  development,  engineering  development,  manufacturing, 
operations,  and  logistics,  and  opportunity  costs 

-  Estimate  impact  of  changing  requirements,  technology,  missions,  and  threats 

-  Identify  the  effects  of  cycle  time  on  each  community,  Warfighters,  Budget, 
Acquisition,  and  Contractor 

-  Identify  under  what  circumstances  cycle  time  reduction  is  advantageous 


Recommendation  1.2  Establish  Clear  Product 
Development  Cycle  Time  Reduction  Goals 

Research  Findings: 

-  No  wide  spread  or  accepted  goal  for  cycle  time  reduction  were  found 

-  No  schedule  performance  or  tracking  metrics  were  found  to  be  used 

-  Project  managers  estimate  that  the  projects  could  be  completed  on  average  in 
50%  of  the  remaining  development  time  without  loosing  capability 

Implications: 

-  Establish  clear  service  and  DoD  wide  goal  for  cycle  time  reduction 

-  Time  from  project  initiation  to  delivery  of  the  first  production  item 

-  Must  establish  current  schedule  baseline  of  all  projects 

-  Separated  into  appropriate  categories  possibly  by  project  type,  size  or  other 
means  so  that  changes  in  each  area  can  be  measured 

-  Goal  must  be  large  enough  to  spur  organizational  change 

-  Recommend  50%  reduction  in  development  time 

-  Stated  goal  of  many  groups,  program  managers  estimate  of  time  required, 
typical  commercial  cycle  time  reduction  goal 

-  Must  specify  time  to  achieve  the  cycle  time  reduction  goal 

•  3  years  typical  commercial  goal 

-  Need  to  collect  schedule  metrics  to  measure  progress  towards  goal 
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' 'Recommendation  1.3:  Appoint  a  Strong  Advocate  for 
Product  Development  Cycle  Time  Reduction 


Research  Findings: 

-  No  organization  or  person  was  found  to  have  been  assigned  or  to  have 
assumed  responsibility  for  cycle  times  or  cycle  time  reduction 

-  Cycle  time  issue  is  submerged  and  lost  during  the  budget  battles 

Implications: 

-  Major  organizational  change  will  not  occur  without  a  significant 
organization  or  person  forcing  the  change 

-  An  strong  organization  must  adopt  or  be  assigned  the  role  of 
advocate  for  cycle  time  reduction 

•  Must  have  a  significant  amount  of  organizational  power 

-  Must  have  method  to  enforce  changes  required  for  cycle  time 
reductions 

-  Suggest  the  acquisition  organizations  embrace  cycle  time 
reduction  as  a  major  organizational  goal  and  become  the  advocate 

-  Use  the  considerable  organizational  power  and  legal  authority  to 
enforce  cycle  time  reduction  across  entire  development  process. 

-  Cycle  time  has  most  direct  impact  on  Acquisition  community 


Recommendation  2 


Develop  and  Use  Rigorous  Schedule 
Based  Information  and  Tools 

Accurate  and  available  information  and  analysis  is 
required  to  make  schedule  based  decisions 

Needs: 

2.1  Require  an  analysis  and  estimate  of  how  long  a  development 
project  “should  take” 

2.2  Develop  effective  project  schedule  estimation  and 
evaluation  tools 

2.3  Require  detailed  schedule  information  on  contractor 
proposed  schedules  for  evaluation 


2.  Develop  and  Use 
Schedule  Based  Information 


Research  Findings: 

-  Little  detailed  schedule  based  information 
available  or  used  to  make  decisions 

-  No  scheduling  tool  were  found  to  be  available 
or  widely  used 

-  Contractors  use  customer  expected  schedule 
as  primary  input  to  develop  their  proposals 

-  Schedule  information  is  difficult  to  evaluate 
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Recommendation  2.1:  Require  an  Analysis  and 

Estimate  of  How  Long  a 
_ _ Development  Project  “Should  Take” 

Research  Findings: 

-  Project  schedules  are  primarily  developed  based  on  expected  funding  availability 

-  There  is  little  technical  rigor  in  development  of  initial  project  schedules 

-  No  systematic  analysis  of  the  required  project  length  based  on  development 
requirements  was  found  for  projects 

Implications: 

-  An  analysis  of  the  required  schedule  duration  to  develop  the  project 
(“Should  Take”  Analysis)  is  needed  early  in  the  planning  of  a  project 

-  Results  of  different  analysis  methods  used  to  evaluate  schedule 

-  Bottoms  up  schedule  development,  parametric  estimation,  analysis  of 
comparable  DoD  projects,  analysis  of  comparable  commercial  projects, 
contractor  estimates 

-  Discussion  of  analysis  and  its  limitations  and  applicability  to  this  project 

-  Identification  of  schedule  drivers  and  associated  schedule  risks 

-  An  analysis  is  needed  of  the  cost  of  the  development  time  to  the  users 
and  of  the  time  based  user  requirements 

-  Cost/benefit  analysis  of  project  based  on  development  time 
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Ideas  to  Include  in 
a  ", Should  Take”  Analysis 


•  Results  of  different  analysis  methods  used  to  evaluate  required  schedule 

-  Contractor  estimates,  Bottoms  up  schedule  development,  parametric  estimation,  analysis  of 
comparable  DoD  projects,  analysis  of  comparable  commercial  projects 

-  Discussion  of  analysis  and  its  limitations  and  applicability  to  this  project 


•  PM’s  Estimate  of  the  Range  of  Time  Required  to  Develop  the  Project 

•  An  Analysis  of  Users  Schedule  Based  Requirements 

-  When  do  they  need  It,  when  do  they  want  It,  and  the  reasons  why 

-  Mission  and  cost  impact  of  not  having  It  by  month/quarter/year 


•  Identify  Current  Schedule  Limiting  and  Risk  Factors 

-  Schedule  sensitivity  analysis  of  requirements  and  other  factors 

-  Resource  loading/availability 


•  Program  Office  Proposed  Schedule  Plans 

-  Proposed  project  initiation  date 

-  Detailed  Proposed  Project  Schedule 

-  Identify  logical  decision  points  and  key  milestones 

-  Identify  Critical  and  Near  Critical  Path  Items 

-  Schedule  Risk  Analysis  and  Sensitivity 

•  Schedule  Management  Approach  Proposed 

-  Project  schedule  goals,  objectives,  and  requirements 

-  Schedule  monitoring  system,  metrics,  and  incentives  to  be  used  at  each  level 
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Recommendation  2.2:  Develop  Effective 
Project  Schedule  Estimation  and  Evaluation  Tools 


Research  Findings : 

-  No  effective  schedule  estimation  tools  or  schedule  related  databases  are  readily 
available  or  used  by  the  Pentagon,  program  offices  or  contractors 


-  Little  impact  of  historicai  project,  parametric  models  bottom  s  up  schedule 
development  on  initial  project  schedule 

-  The  initial  project  schedule  essentially  determines  the  minimum  development  time 


Implications: 

-  Need  a  comprehensive  data  base  of  project  schedule  information 

-  Collect  current  project  schedules  at  a  level  detailed  enough  for  comparison 

-  Develop  detailed  schedule  guidelines  and  benchmarks. 

-  Suggest  use  a  detailed  schedule  based  surveys  of  all  ongoing  projects 

-  Augment  with  commercial  product  development  databases  and  experiences 

-  Need  to  develop  advanced  schedule  estimation  and  risk  analysis  tools 

-  Need  to  be  able  to  effectively  determine  risk  in  proposed  schedules 

-  Suggest  control  of  schedule  models  stay  with  the  acquisition  community 

-  Must  be  readily  available  to  PM  for  regular  and  easy  use  for  early  planning 
and  ongoing  project  schedule  management 

•  Maintain  schedule  models  distinct  but  compatible  with  existing  cost  models 
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Recommendation  2.3:  Require  Detailed 
Information  on  Contractor  Proposed  Schedule 


Research  Findings: 

-  Contractors  primary  schedule  input  is  customers  expected  schedule  and 
analytical  tools  and  analysis  do  not  significantly  effect  the  proposed  length 

-  Contractor  proposed  schedule  related  information  is  currently  spread  across 
different  proposal  sections  and  is  often  not  well  coordinated  or  complete 

-  Government  members  report  to  view  any  alternative  schedule  different  from  the 
government  expected  schedule  with  suspicion  and  presumes  a  higher  risk. 

Implications: 

-  Additional  information  needed  to  effectively  evaluate  the  contractors 
ability  to  execute  proposed  schedules  and  the  associated  schedule  risk 

-  Suggests  some  sort  of  “Schedule  Proposal”  that  details  and  explains  the 
proposed  schedule  and  the  justification  for  it. 

-  Justification  of  proposed  schedule  with  analysis  of  the  projected  risks 

-  All  critical  and  near  critical  paths  identified,  and  discussion  limiting  factors 

-  Company  demonstrated  schedule  capability  and  past  performance 

-  Should  be  structured  to  allow  consistent  and  effective  schedule  based 
evaluation  of  the  capability  and  the  risk  for  each  company 

-  Proposed  methods  to  manage,  measure,  and  provide  incentives  for  schedule 


Recommendation  3 


Provide  Incentives  That 
Encourage  Cycle  Time  Reduction 

Aligning  personal  and  organizational  incentives  with 
cycle  time  reduction  will  be  essential 


LEAN  AEROSPACE 
INITIATIVE 


Needs: 

3.1  Provide  incentives  for  Cycle  Time  Reduction  Within  the  Government 

3.2  Provide  Incentives  for  Cycle  Time  Reduction  for  Contractors 

3.2.1  Make  the  length  of  the  development  schedule  and  the 
associated  risk  a  significant  source  selection  criteria 

3.2.2  Provide  significant  schedule-based  contract 
incentives  to  meet  and  reduce  the  schedule 

3.3  Provide  a  Widespread  Training  on  Best  Product  Development 
Practices 


3.  Provide  and  Align  Incentives 
For  Cycle  Time  Reduction 


Research  Findings: 

-  No  incentive  reported  by  any  level,  Pentagon, 
Program  office,  or  contractor,  to  reduce 
development  cycle  time. 

-  Shortened  schedule  last  of  4  project  objectives 

-  Many  projects  completed  exactly  on  schedule 

-  Very  few  projects  ever  completed  ahead  of 
schedule 

-  Schedule  first  to  be  sacrificed 
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Recommendation  3.1:  Provide  Incentives  To 
Reduce  Cycle  Time  at  Program  Office  and  Pentagon 


Research  Findings: 

-  Little  personal  accountability  or  feeling  of  responsibility  for  schedule 

-  Many  project  managers  per  project  3.7  average  -  Current  avg.  time  15  months 

-  Shortening  schedule  seen  as  lowest  priority  at  Program  Office  and  Pentagon 

Implications: 

Need  personal  and  organizational  incentives  for  program  offices  and 
Pentagon  levels  for  cycle  time  reduction 

-  Develop  and  track  schedule  based  metrics  for  all  projects  to  track  performance 

-  Increase  program  office  and  project  manager  schedule  accountability 

-  Assign  project  and  program  managers  to  the  projects  for  3  years  minimum 
periods 

-  Use  schedule  length  and  schedule  performance  measures  as  a  significant 
performance  metric  for  program  office  and  personal  performance  ratings. 

-  Make  project  cycle  time  and  cycle  time  reduction  a  significant  part  of  program 
office  and  personal  award  programs 

-  Use  development  time  as  a  significant  factor  in  the  project  screening  to  select 
which  projects  process  to  the  next  phase. 
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Recommendation  3.2.1  Make  schedule 
development  schedule  and  schedule  risk  a 
_ source  selection  catena 

Research  Findings: 

-  Program  Managers  report  schedule  is  not  an  important  source  selection  criteria 

-  Government  expected  schedule  contractors  primary  schedule  input. 

-  Contractors  report  no  incentive  to  reduce  cycle  timer  to  bid  a  shorter  schedule 

-  Very  few  contractors  propose  schedules  other  than  government  expected  schedule 

-  Very  few  projects  ever  complete  ahead  of  the  contracted  schedule 

Implications: 

-  Schedule  development  time  and  the  associated  schedule  risk 
should  be  a  more  significant  source  selection  criteria 

-  Establish  effective  schedule-based  source  selection  criteria 

-  Use  previously  discussed  schedule  proposal  as  basis  for  evaluation. 

-  Develop  effective  and  accepted  schedule  evaluation  procedures 

.  Use  development  time  as  a  significant  criterion  at  each  stage  of  the  project 
screening  process  used  to  manage  the  portfolio  of  projects. 

-  Combine  with  significant  schedule  based  incentives  meet  or  shorten  schedules 


LEAN  AEROSPACE 
INITIATIVE 


Recommendation  3.2.2  Provide  Significant 
Contract  Incentives  for  the  Reduction  of 
_ Development  Schedules 


Research  Findings: 

-  Contractors  report  little  or  no  Incentive  to  accelerate  contract  schedules 

-  Program  offices  rept  rt  no  contract  incentives  to  meet  or  reduce  project  schedules 

-  Few  project  schedules  are  ever  accelerated 

-  Shortened  schedule  reported  as  lowest  project  priority  at  all  levels 

Implications: 

-  Need  to  provide  significant  schedule  based  incentives  to  the  contractor 
to  meet  and  also  reduce  project  schedules  in  the  development  phase. 

-  Need  an  approved  framework  schedule  based  contract  incentives. 

-  Needs  an  accepted  and  standardized  menu  of  options  for  schedule  based 
incentives  which  are  acceptable  and  for  which  the  implications  and 
implementation  are  understood  by  both  government  and  contractors. 

-  Incentive  structures  must  encourage  meeting  and  reducing  the  schedules. 

-  Incentives  must  be  structured  so  as  not  to  undermine  other  project  goals 

-  The  government  should  consider  the  effect  of  proposed  schedule  based 
incentives  as  part  of  the  evaluation  of  the  proposals  and  its  risk. 

-  Suggests  contractors  propose  the  specific  incentives  structure  and  levels 

that  they  would  like  as  part  of  their  proposal. _ 


Recommendation  3.3  Provide  Widespread 
Training  on  Best  Product  Development  Practices 


Research  Findings: 

-  There  is  little  awareness  of  best  product  development  practices  in  other  industries 

-  Only  lof  5  SPO  Project  Managers  have  taken  a  course  in  product  development 

-  Government  acquisition  courses  appear  to  teach  current  DoD  policy  and  do  not 
teach  best  product  development  practices  throughout  other  related  industries 

Implications: 

-  Require  a  comprehensive  training  program  to  educate  acquisition 
personal  on  the  best  product  development  practices  from  all  industries 

-  Suggests  a  multifaceted  training  program  poqsibly  including: 

-  Individual  Efforts  -  Establish  a  reading  program  with  discussion  groups. 

-  SPO  and  Center  Level  Training  -  Establish  recurring  seminars  on  best  product 
development  practices  with  speakers  from  variety  of  industries  -  Use 
Commercial  Product  Development  Consultants 

-  Formal  Training  -  Establish  lessons  in  current  training  programs  on  best 
product  development  practices  based  on  other  industries  experience.  -  Bring  in 
Commercial  Product  Development  Consultants 

-  Formal  Education  -Establish  a  competitive  scholarship  program  for  officers  to 
attend  top  academic  institutions  to  study  best  product  development  practices 


C»p<  McNutl  PO  '1988  Mitslchu&ftU  Institute 


LEAN  AEROSPACE 
INITIATIVE 


Recommendation  4 


Mitigate  Funding 
Based  Schedule  Limitations 

The  available  resources  must  be  allocated  across  the 
portfolio  of  projects  and  made  available  in  a  manner 
to  allow  for  cycle  time  reduction 

Needs: 

4.1  Require  New  projects  be  fully  funded  based  on  development  requirements 

4.2  Develop  an  effective  screening  process  to  manage  the  portfolio  of  projects 

4.3  Limit  the  number  of  projects  in  each  phase  of  development 

4.4  Clear  the  log  jam  of  current  development  projects 

4.5  Make  Available  resources  to  accelerate  projects  as  opportunities  arise 
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4.  Mitigate  Funding 
Based  Constraints 


•  Research  Findings: 

-  80%  of  projects  schedules  report  to  be  funding  constrained 
not  technically  constrained 

-  Estimated  time  required  to  completion  average  50%  of 
current  planned  time  to  completion. 

-  Most  initial  project  plans  based  on  funding  constraints  not 
technical  requirements 

-  Inability  to  locate  funding  cited  as  most  important  barrier  to 
cycle  time  reduction 
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Recommendation  4.1  Require  projects  be  fully  funded 
based  on  development  related  requirements 


Research  Findings: 

-  Current  milestone  approval  practice  requires  project  be  “fully  funded"  based  on 
the  funding  requirements  “as  planned”,  not  necessarily  what  is  optima! 

-  Initial  project  plans  were  shown  to  be  based  primarily  on  expected  funding 
availability  and  not  based  on  the  development  requirements. 

-  80%  of  project  schedules  are  reported  limited  by  available  funding  and  not  limited 
by  technical  development  aspects. 

-  PM  estimate  they  could  finish  development  in  50%  of  remaining  development  time 

Implications: 

-  Project  funding  profiles  must  be  based  on  the  time  and  resources 
required  to  develop  the  project  •  not  the  perceived  funds  available 

-  Develop  optimal  schedule  -  then  allocate  funds  required  through  PPBS 
process  to  carry  out  the  project. 

-  Limit  total  amount  of  resources  required  by  limiting  number  of  projects 

-  Need  to  manage  the  portfolio  of  projects  allow  optimal  funding  of  the 
projects  in  development 

-  Funding  profiles  must  be  maintained  after  a  milestone  decisions 
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Recommendation  4.2  Develop  an  effective 
project  screening  process  to  manage 
_ the  portfolio  of  projects 

Research  Findings: 

-  The  funding  and  PPBS  process  is  not  effective  process  to  control  projects  entering 
into  each  phase  of  development  to  ensure  each  is  optimally  funded 

-  Current  limiting  process  primarily  based  on  ability  to  locate  funds  to  initiate  project 

-  Milestone  decision  process  not  effectively  limiting  number  of  projects  (esp.  small) 

-  Only  58%  of  projects  are  stated  due  to  modernization  planning  process 

Implications: 

-  Needs  a  form  of  competitive  project  screening  process  to  allow  the  highest 
priority  projects  that  are  fully  ready  and  can  be  supported  by  the  available 
resources  to  enter  each  phase  of  development. 

-  Select  the  projects  that  will  be  allowed  to  proceed  to  the  next  development  phase, 
those  that  will  remain  In  the  current  phase,  and  those  that  will  be  canceled. 

-  Require  all  projects  to  pass  through  the  process  (allow  very  few  exceptions) 

-  Senior  leaders  to  run  a  form  of  a  “Project  Evaluation  and  Selection  Board" 

-  Direct  project  to  project  comparisons  -  Similar  to  Office  Promotion  Board 

-  Selection  categories  by  categories  that  could  include  mission  area  and  ACAT  level 
or  some  other  logical  grouping 

-  Use  results  of  selection  process  as  direct  input  into  PPBS  process. 


Proposed  Project  Screening  Process  Structure 


Pre-qualification 
Certification  and  Ratings 


•  Pre-qualification  states  that,  if  approved,  the  project  stands  a  very 
high  probability  of  completing  within  its  cost,  schedule  and 
performance  requirements  and  that  it  will  meet  the  operational 
mission  that  it  is  intended 

•  Pre-Qualification  Certifications  -Ensure  that  the  estimates  in  each 
area  are  accurate  as  stated  and  level  of  risk  identified 

-  Projected  Cost  -  FM  Responsibility 

-  Operational  -  Users  Responsibility 

-  Schedule-  AQ  Responsibility 

-  Management  Plan  -  AQ  Responsibility 

-  Technology  Proven  -  AQ  Responsibility 

•  Contract  Options  Signed  and  Ready  (Milestone  II  and  III) 

•  Pre-qualifications  certification  and  ratings  would  be 
obtained  from  the  respective  organizations  prior  to  the 
Project  Selection  Board.  No  project  would  be  allowed  to 
meet  the  selection  board  without  them. 
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Project  Selection  Board 


Purpose  to  recommend  or  select  the  projects 
that  will  enter  each  phase  of  development 

•  Formed  by  Senior  Service  Leaders  or  their  designees 

•  Would  do  a  project  by  project  comparison  within  certain  categories 

-  Project  areas  separated  by  size  and  possibly  mission  areas 

•  Projects  would  be  rank  ordered  based  on  information  presented  within 
each  area. 

•  Senior  leaders  would  select  projects  based  on  criteria  would  be  based 
on  budget  allocation. 

•  Final  selection  made  by  Service  and  DoD  leadership 

•  Have  option  of  returning  projects  to  current  phase,  completing  and 
documenting  or  selecting  it  for  the  next  phase. 
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Recommendation  4.3  Limit  the  number  of 
projects  in  each  phase  of  development 

Research  Findings: 

-  No  established  limit  to  the  number  of  projects  in  the  each  development  phase 

-  No  effective  control  method  -  Ability  to  obtain  and  maintain  funding  only  limit 

-  Little  control  or  incite  over  smaller  projects  -  too  many  projects  for  leaders  to 
effectively  oversee  even  at  program  office  level 

Implications: 

-  Need  to  establish  firm  limits  and  controls  on  the  number  of  projects 
allowed  in  each  phase  of  development  at  any  one  time. 

-  Decreasing  number  of  projects  per  phase. 

•  Force  selection  be  make  hard  decisions  between  projects. 

-  Encourage  to  push  projects  through  the  development  process  or  stop  them 

-  Establish  limits  based  on  the  amount  of  projected  funding  with  reserve 

-  Should  specify  mix  of  size,  projects,  weapon  system  types 

-  Allow  trades  small  Vs  large  projects  with  an  appropriate  weighting  (i.e.  7 
small  to  1  large) 

-  Allocate  number  of  projects  prior  to  selection  board  with  opportunity  to  revisit 
following 
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Recommendation  4.4  Clear  log  jam 
of  current  projects 


Research  Findings: 

-  Log  jam  of  current  projects  in  development  is  causing  large  amount  of  project  slip 

-  Funding  Instability  accounting  for  half  of  project  slips 

-  Number  of  individual  projects  in  development  but  the  actual  number  is  unknown. 

-  Recent  large  reduction  in  available  development  funds  -  few  project  cancellations 

Implications: 

-  Need  to  address  the  large  number  of  projects  that  are  currently  in 
development 

-  Must  identify  alt  current  development  projects  at  the  project  level 

-  Categorize  into  project  types  and  phases  for  project  screening  process 

-  Identify  the  lowest  priority  projects  and  stop  them  -  use  resources  to 
accelerate  others 

-  Time  phase  remaining  development  projects  to  conduct  a  significant  number 
in  serial  and  not  parallel  allowing  resources  to  be  concentrated 


PO  1/2V98-29  •  1998 


LEAN  AEROSPACE 
INITIATIVE 


Recommendation  4.5  Ensure  Necessary  Resources 
are  Available  to  Accelerate  Projects  schedules 


Research  Findings: 

-  Lack  of  available  funds  reason  most  cited  for  not  accelerating  projects 

-  Very  few  projects  ever  accelerated 

-  Project  managers  estimate  projects  could  be  done  in  half  the  remaining  time 

Implications: 

—  Need  some  for  of  management  reserve  that  can  be  used  for  cycle  time 
reduction  efforts 

-  Give  cycle  time  reduction  requests  priority  for  available  funds 

-  Earmark  fallout  funds  for  cycle  time  reduction  efforts  first 

*  Disallow  use  of  fallout  money  to  be  used  for  new  starts  or  additional 
requirements 

-  New  starts  should  be  controlled  by  a  centralized  project  selection  process 

-  allow  Product  Centers  and  PEO  to  use  available  funds  within  their  areas  first 
for  cycle  time  reduction,  then  allow  other  PEO  and  Product  Centers  to  use  any 
remaining  funds  for  cycle  time  reduction  efforts,  prior  to  initiating  any  new 
projects  or  project  feature  or  requirement  additions. 
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Recommendation  5 


Demonstrate  Application  and  Results 
Through  Near  Term  Demonstration  Projects 

A  demonstrated  capability  and  impact  of  cycle  time 
reduction  efforts  is  needed  at  both  the  project  and 
portfolio  of  projects  level  before  full  implementation 

Needs: 

5.1  Execute  Pathfinder  projects  that  demonstrate  the  effectiveness  of  the 
cycle  time  reduction  approach  on  specific  projects  (Project  Based 
Demonstration) 

5.2  Execute  a  Re-invention  Laboratory  that  demonstrates  effectiveness  of 
approach  on  a  range  of  products  within  an  entire  program  office  or 
product  center  (Portfolio  of  Projects  Based  Demonstration) 
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5.  Demonstrate  Application 
Through  Demonstration  Projects 


Research  Findings: 

-  Very  few  project  ever  completed  early 

-  Funding  based  issues  are  systematic  across  all  projects 

-  Examples  of  similar  commercial  projects  finishing  in 
significantly  less  time. 

-  Commercial  constraints  are  significantly  different  then  in 
defense  industry 

-  Few  previous  efforts  to  reduce  programmatic  limitations  on 
product  development  cycle  time  from  which  to  base 
decisions 

-  Few  demonstrated  examples  to  show  a  proper  path  to 
reduce  cycle  time 
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Recommendation  5.1  Execute  Pathfinder  projects 
that  demonstrate  the  ability  to 
_ reduce  cycle  time  on  specific  projects 


Research  Findings: 

-  Few  examples  of  cycle  time  reduction  -  No  clear  demonstrated  capability  to 
shorten  times 

-  Wide  spread  skepticism  about  impact  of  cycle  time  reduction  on  cost  and 
performance. 

Implication: 

-  Select  and  run  a  series  of  path  finder  projects  that  demonstrate  the  ability  to 
reduce  cycle  time  on  a  wide  array  of  project  types. 

-  Select  a  range  of  new  and  existing  projects  from  all  product  centers  as  pathfinder 
projects  to  show  results  of  a  cycle  time  focus  and  cycle  time  reduction  efforts. 

-  Make  direct  comparisons  with  similar  “control”  projects  not  selected  for  the 
pathfinder  status  or  cycle  time  focus 

-  Capture  data  on  results  reported  by  all  effected  organization, 

-  Compare  to  previously  planned  and  resulting  pathfinder  approaches. 

-  Capture  lessons  learned  and  problems  faced  at  the  project  level. 

-  Allow  Pathfinders  projects  wide  latitude  to  take  actions  they  feel  are  appropriate 
to  reduce  cycle  time. 
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Recommendation  5.2:  Execute  A  Re-invention 
Laboratory  to  Demonstrate  of 
_ the  Effect  Over  a  Portfolio  of  Projects 

Research  Findings: 

-  Product  development  cycle  time  effects  not  only  each  project  but 
effects  the  development  process  as  a  whole 

-  The  focus  on  cycle  time  in  the  commercial  industry  has  been  on 
across  a  range  of  projects  and  not  on  one  single  projects 

-  No  demonstration  has  been  conducted  to  account  for  compounding 
effects  of  cycle  time  reduction  on  a  set  of  projects 

implications: 

-  Need  a  number  of  program  offices  with  many  development  projects  to 
act  as  Re-invention  laboratories  to  demonstrate  the  compounding 
impact  of  cycle  time  focus  on  a  portfolio  of  projects  over  time. 

-  A  basket  SPO  or  a  mature  development  project  would  have  a  large 
range  of  projects  could  demonstrate  the  effectiveness  of  this 
approach  within  a  controlled  environment. 

-  A  matching  program  office  without  the  re-invention  lab  could 
provide  a  control  comparison  to  demonstrate  effect  of  changes. 
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Near  Term  Key  Issues 


•  Provide  answers  to  the  following  questions: 

-  Should  cycle  time  reduction  be  an  element  of  acquisition  reform? 

-  What  are  the  barriers  and  the  benefits? 

-  What  are  the  best  methods  to  achieve  cycle  time  reductions? 

•  Develop  tools  to  help  the  acquisition  workforce  implement  the 
required  changes 


Near-Term  Steps 


Address  The  Following  Recommendations: 

1.1  A  Clear  Business  Case  Or  Rationale  for  Cycle  Time  Reduction 

2.1  Analysis  and  Estimate  of  How  Long  a  Development  Project 

“Should  Take” 

2.2  Develop  Effective  Project  Schedule  Estimation  and 

Evaluation  Tools 

3.3  Establish  an  Education  Program  on  Best  Practices  in  Product 

Development 

4.1  Pathfinder  Projects  That  Demonstrate  the  Effectiveness  of  the 

Cycle  Time  Reduction 

4.2  Re-Invention  Laboratory  That  Demonstrates  Effectiveness  of 

Approach  on  a  Portfolio  of  Products 
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'Appendix  2 

Impact  of  Long  Development  Times 

Long  development  times  impact  the  DoD  in  many  ways.  It  impacts  our  military 
capability  through  systems,  long  in  development,  not  being  ready  when  needed.  It 
impacts  our  military  capability  through  systems  not  meeting  the  current  need  when 
fielded.  It  impacts  our  military  capability  through  fielding  of  dated  technology  in  our 
newest  systems.  It  impacts  our  ability  to  quickly  respond  to  new  or  emerging  threats  or 
to  respond  to  known  safety  issues.  Long  development  times  also  result  in  increased  cost 
to  develop  and  sustainment  of  our  weapon  systems.  Examples  of  each  type  of  impact  of 
long  development  times  are  provided  below. 

A.  Systems  Not  Ready  When  Needed 

Desert  Storm  provided  a  unique  opportunity  to  identify  military  needs  in  a 
wartime  environment  and  determine  the  possible  impact  of  equipment  in  the  development 
pipeline  had  it  had  been  deployed  more  quickly.  Seven  systems  would  have  mitigate 
critical  needs  during  the  early  part  of  Desert  Shield  and  Desert  Storm. 

Long-Range,  Wide-Body  Strategic  and  Tactical  Airlift:  The  C-17 

In  the  late  1970s,  the  Air  Force  identified  a  need  for  the  C-17  to  supply  the  newly 
formed  U.S.  Rapid  Deployment  Force.  The  request  for  proposals  for  the  C-X  cargo  plane 
was  released  in  October  1980.  Eleven  years  later,  during  the  earliest  stages  of  Desert 
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Shield  and  the  Gulf  War,  a  critical  shortage  of  heavy-lift  aircraft,  combined  with  a  lack  of 
suitable  airfields  in  Saudi  Arabia,  limited  the  speed  by  which  the  military  buildup  could 
occur.  This  was  precisely  the  mission  for  which  the  C-l  7  was  being  developed  and 
designed.  The  C-l 7  was  delayed  in  part  owing  to  a  decision  to  produce  more  C-5B 
aircraft.  The  C-l 7  did  not  see  its  first  operational  use  until  Operation  Provide  Hope  in 
Bosnia  in  1995."5 

Satellite  Communications:  MILSTAR 

Development  of  the  MILSTAR  satellite  system,  designed  to  upgrade  the  military 
communication  system,  began  in  1981.  In  1983,  the  Air  Force  issued  a  contract  designed 
to  provide  secure  and  survivable  tactical  and  strategic  communication  by  the  late  1980s. 
Eight  years  later,  the  communication  capacity  required  to  conduct  the  early  stages  of 
Desert  Shield  proved  inadequate,  as  existing  UHF  and  SHF  systems  were  susceptible  to 
jamming  and  interference.  Communication  officers  finally  cobbled  together  an  adequate 
communication  system  by  employing  leased  commercial  satellites,  repositioning  existing 
military  satellites,  and  reviving  decommissioned  communication  satellites.116  The  first 
MILSTAR  satellite  was  not  launched  until  February  1994,  and  the  second  satellite  not 
until  November  1995. 

Precision  Targeting:  LANTIRN  Targeting  Pod 

The  Low  Altitude  Navigation  and  Targeting  Infrared  at  Night  (LANTIRN)  system 
was  initiated  in  1979  to  give  F-16s  and  F-15s  the  ability  to  navigate  at  night  and  to  lock 
onto  targets  precisely  and  use  precision  guided  weapons.  During  Desert  Storm,  1 1  years 
into  a  12-year  development  effort,  only  six  targeting  pods  were  available  for  F-15Es  and 
no  targeting  pods  were  available  for  F-16s.  This  left  most  deployed  fighters  without 
precision  bombing  capability.  This  increased  number  of  aircraft  required  to  attack  each 
target,  decreased  the  lethality  of  the  attack,  decreased  the  distance  from  the  target  the 


1.5  Information  provided  by  the  C-17  Program  Element  Monitor  and  several  Aviation  Week  and  Space 
Technology  articles  from  the  Early  1980. 

1.6  Alan  Campen  Ed.  The  First  Information  War.  Armed  Forces  Communications  and  Electronics 
Association  Fairfax  Virginia.  October  1992.  Pg.  8 
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pilots  could  release  the  bombs,  and  exposed  pilots  to  greater  risk.  Immediately  after 
Desert  Storm,  the  targeting  pods  were  produced  and  fielded  at  a  rate  of  28  per  month.  By 
1992, 400  F-16s  were  outfitted  with  the  LANTIRN  system,  which  had  the  development 
time  been  shorter  may  have  been  available  for  use  in  desert  storm."7 

Secure  Data  Links:  Joint  Tactical  Information  Distribution  System 

The  Joint  Tactical  Information  Distribution  System  (JTIDS)  program  was  initiated  in 
1974  to  enable  all  U.S.  military  fighter  and  command-and-control  aircraft  to  share  data  on  the 
location  of  friendly  and  enemy  aircraft  and  targets.  During  Desert  Storm,  19  years  into  a  20- 
year  development  effort,  no  fighter  aircraft  were  equipped  with  JTIDS.  The  results  was  less 
situational  awareness  for  U.S.  pilots,  higher  risk  of  fratricide,  and  the  possible  escape  of 
additional  Iraqi  planes  to  Iran.ll8The  system  was  first  deployed  in  1993  on  the  F-15s  at 
Mountain  Home  AFB.  In  a  matter  of  weeks,  the  fighter  wing  employed  new  tactics,  using  a 
line  formation,  that  allowed  a  dramatic  increase  in  effectiveness  due  directly  to  the  increase  in 
situational  awareness  provided  by  JTIDS."9. 

Satellite  Navigation:  The  Global  Positioning  System 

The  satellite  Global  Position  System  (GPS)  was  initiated  in  December  1973  to  provide 
continuous  worldwide,  three-dimensional  navigation  ability  to  U.S.  air,  ground,  and  sea  forces. 
At  the  beginning  of  Desert  Storm,  21  years  into  a  23-year  development  effort,  GPS  could 
provide  only  14  hours  of  navigation  daily.  Very  few  GPS  receiver  equipment  was  available  to 
operators  of  aircraft,  vehicles,  or  to  soldiers  for  personal  user.  Soldiers’  parents  purchased 
commercially  available  GPS  receivers  and  sent  them  to  their  sons  and  daughters.  The  Japanese 
purchased  American-made  commercial  GPS  receivers  for  U.S.  military  personnel.  By  the  end 
of  Desert  Storm,  the  launch  of  additional  satellites  provided  23  hours  of  navigation  coverage, 
but  many  military  units  still  did  not  have  user  equipment.  The  lack  of  GPS  field  receivers 
meant  that  more  soldiers  were  lost  in  the  desert  and  captured.  The  U.S.  Army  also  attributed 


1,7  Information  provided  by  the  LANTIRN  Program  Office. 

118  Information  provided  by  a  former  program  director  for  JTIDS. 

119  Information  provided  by  a  former  program  director  for  JTIDS. 
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most  fratricide  incidents  to  armored  units  that  were  lost  and  out  of  position.120  Because  of  the 
lack  of  available  military  receivers  that  could  use  the  encrypted  GPS  signals,  the  U.S.  had  to 
provide  the  GPS  signals  without  encryption  and  as  a  results  did  not  have  the  capability  of 
denying  the  use  of  GPS  system  to  Iraq  while  using  the  signal  itself.121 

The  Advanced  Medium-Range  Air-to-Air  Missile 

The  Advanced  Medium-Range  Air-to-Air  Missile  (AMRAAM)  was  initiated  in 
November  1978  to  give  U.S.  fighter  aircraft  an  advanced  radar  system  for  guiding  missiles. 
AMRAAM  was  designed  to  provide  a  multiple-target,  fire-and-forget  capability  at  extended 
range,  expanding  the  engagement  envelop  over  the  Vietnam-era  AIM-7  Sparrow  missile. 
During  Desert  Storm,  12  years  into  a  13-year  development  effort,  no  operational  AMRAAMs 
were  available,  although  several  missiles  were  used  in  tests.  After  Desert  Storm  testing  was 
accelerated,  and  operational  capability  was  declared  six  months  later,  in  September  1991. 

Wide-Area  Anti-Tank  Cluster  Bomb:  The  CBU-97B  Sensor  Fuse  Weapon 

Development  of  the  Sensor  Fuse  Weapon  (SFW)  was  initiated  in  January  1983,  to 
provide  the  ability  for  a  limited  number  of  aircraft  to  destroy  up  to  24  tanks  in  a  single 
attack.  During  Desert  Storm,  8  years  into  a  14-year  development  program,  no  Sensor  Fuse 
Weapons  were  available.  The  U.S.  thus  had  limited  capability  to  stop  an  early  Iraqi  armored 
invasion  of  Saudi  Arabia  had  one  occurred,  and  additional  aircraft  sorties  were  required  to 
attack  Iraqi  armor,  increasing  the  risk  to  pilots  and  destroying  fewer  Iraqi  tanks.122 

Many  other  major  defense  systems  under  development  for  at  least  five  years  were 
not  available  for  use  in  Desert  Shield  or  Desert  Storm.  Those  included  the  Stingray  Anti- 
Aircraft  Missile,  the  V-22  Osprey,  the  AGM-130  Powered  Glide  Bomb,  the  Mark  XV 
Identification  Friend  or  Foe,  the  Army  Brilliant  Anti-Tank  Weapon,  the  Advanced 
Apache  Long  Bow  and  Hellfire  Missile  System,  the  Commanche  attack  helicopter,  the 
AWACS  Block  30-35  upgrade  program,  the  F-22  Air  Superior  Fighter,  and  the  B-2 


120  U.S.  Army  report  on  GPS  in  Desert  Storm. 

121  Information  provided  by  the  GPS  Program  Office. 
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strategic  bomber.  All  these  systems  were  started  based  on  an  identified  need.  Many  of 
these  systems  have  still  not  reached  operational  status  7  years  later. 

A  few  systems  were  rushed  through  development  and  made  available  to  troops 
during  the  six  months  prior  to  Desert  Storm.  The  Joint  STARS  surveillance  plane  was 
pushed  into  service  and  provided  critical  observations  of  Iraqi  troop  movements.  A  bomb 
system,  the  Bunker  Buster,  was  developed  and  fielded  in  29  days.  This  rapid  action 
demonstrated  that  the  acquisition  system  can  move  quickly  to  meet  the  needs  of 
warfighters  when  they  are  seen  as  essential.  But  such  efforts  are  the  exception  rather  than 
the  rule. 


B.  Systems  Not  Meeting  Current  Needs  When  Fielded 

With  the  average  development  time  for  a  new  major  defense  system  approaching 
10  years,  the  need  for  and  requirements  of  any  system  in  development  may  dramatically 
change.  Many  of  the  systems  now  in  the  pipeline  are  based  on  the  threat  and  political 
environment  that  existed  before  the  dissolution  of  the  Soviet  Union  and  the  Gulf  War.  Of 
26  current  major  development  programs  in  the  Rand  database  and  due  for  completion 
between  1995  and  1999,  only  6  were  started  following  the  end  of  the  Cold  War.  Twelve 
began  during  the  early  1980s,  when  the  U.S.  faced  a  radically  different  environment.  One 
result  is  that  systems  often  do  not  adequately  meet  warfighters  current  needs  when 
fielded. 

MILSTAR I 

Communication  is  a  significant  area  where  new  systems  are  not  meeting  today’s 
requirements.  During  the  early  1980s,  for  example,  the  most  pressing  need  was  seen  as 
communication  systems  that  could  survive  a  full  range  of  conflicts,  from  regional  to  full- 
scale  nuclear  war,  and  also  foil  enemy  attempts  to  jam  the  signals.  In  response  to  these 


122  Information  provided  by  the  Wide  Area  Munition  program  rlement  monitor  and  the  Wide  Area 
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needs,  the  MILSTAR  program  was  started  in  1981.  But  by  the  time  the  first  satellite  was 
deployed  in  1994,  the  requirements  had  changed  dramatically.  In  the  Gulf  War,  the  most 
pressing  problems  were  not  security  and  anti-jam  capabilities  but  bandwidth  and  the 
capacity  to  process  data  quickly.  Even  if  MILSTAR  had  been  operational  in  time  for 
Desert  Storm,  its  low  data-processing  rate  would  not  have  helped  meet  the  large  demand 
for  fast  communication.  A  significant  portion  of  communication  today  is  in  the  form  of 
digital  data  and  fax  transmission,  which  MILSTAR’s  2400-baud  data  rate  does  not 
effectively  support.  The  development  of  MILSTAR  consumed  most  of  the  available 
resources  to  upgrade  and  maintain  other  military  satellite  communication  systems.  The 
MILSTAR  II  satellite  now  in  development  is  projected  to  increase  the  data-processing 
rate  moderately,  to  9600  baud,  but  even  this  new  design  will  not  be  able  to  support 
communication  needs  as  they  existed  during  Desert  Storm.123 

C.  New  Systems  Fielded  with  Dated  Technology 

In  the  time  now  required  to  develop  and  field  a  new  military  system,  technology  is 
no  longer  state  of  the  art,  and  in  some  cases  it  is  obsolete  and  out  of  production.  Ten  year 
ago,  the  Packard  Commission  stated  that  long  development  cycles  “  lead  to  obsolete 
technology  in  our  fielded  equipment.  We  forfeit  our  five-year  technological  lead  by  the 
time  it  takes  us  to  get  our  systems  from  the  laboratory  into  the  field.” 124  This  problem  is 
severely  exacerbated  by  the  rapid  rate  of  advance  in  electronics.  Technologies  are  usually 
selected  and  “frozen”  early  in  full-scale  development,  significantly  undermining 
programs  based  on  fast-moving  electronic  and  computing  technologies.  These  effects  can 
be  seen  in  programs  such  as  the  F-22,  Joint  STARS,  and  AW ACS  Radar  System 
Improvement  programs  (RSIP),  all  of  which  are  heavily  dependent  on  computer 
processing.  Current  processors  available  operate  significantly  faster.  As  a  point  of 
referenceof  today’s  technology  used  in  home  computers,  the  current  Pentium  II  processor. 


Munition  Program  Office. 

123  Campen,  Alan  Editor.  The  First  Information  War:  The  Story  of  Communications,  Computers,  and 
Intellegence  Systems  In  the  Persian  Gulf  War.  Fairfax  Virginia:  AFCEA  International  Press.  1992.  Pg.  58 
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operating  at  300MHz,  can  execute  the  equivalent  of  627  million  instructions  per  second 
(MIPS).125 

AWACS  RSIP 

The  initial  computer  processor  for  the  AWACS  Radar  System  Improvement 
Program,  selected  in  1986,  was  the  most  advanced  then  available--a  specially  modified  R- 
3000  at  10-12  million  instructions  per  second.  In  1995,  the  production  of  the  modified  R- 
3000  was  cancelled  so  the  processor  was  upgraded  to  a  specially  modified  R-4400  at 
running  at  33-50  MIPS-at  an  additional  cost  of  $26  million.  In  1998,  the  manufacturer 
again  stopped  producing  the  modified  processor,  and  the  Air  Force  made  a  $4  million 
lifetime  buy  to  satisfy  all  its  future  needs  for  the  processors  for  the  expected  life  of  the 
equipment.  Now  the  service  wants  to  upgrade  to  the  Power  PC  processor  to  allow  for 
additional  capability,  but  no  funds  are  available  to  pay  for  another  upgrade.126  The  current 
processor  is  running  at  near  capacity  with  the  existing  software  modifications  leaving 
little  room  for  additional  functions. 


124  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “  A  Formula  For  Action:  A  Report  to 
the  President  on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986. 

125  Based  on  analysis  of  processor  performance  from  Intel  Corporation  processor  facts  sheets  from  their 
web  pages. 

126  Source:  AWACS  Program  Office 
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30  proc  p*r  pUn*  .  10  for  EW  sys 
Processor «nd  of  production  1*91. 
Processor  upgrade  start  00  and  06 
Not  included  in  first  5  prod  lots. 


Figure  A2-1:  The  Progression  of  Commercial  Computer  Processors  vs.  Those 
Fielded  in  AWACS  RSIP,  Joint  STARS  and  F-22  Aircraft 127 


Joint  STARS 

The  processor  for  the  Joint  STARS  Main  Mission  computer  was  selected  in  1991 
as  the  DEC  VAX  66D,  running  at  56  MIPS.  The  system  required  three  active  computers, 
three  to  run  mission  software  and  two  to  operate  as  ready  spares.  But  the  original 
processor  is  out  of  production.  An  upgrade  program  was  started  in  1996  that  will  replace 
the  five  computers  with  one  operating  and  one  spare  processor,  to  be  included  in  aircraft 
delivered  in  2005  and  later.  The  current  fleet  will  also  be  upgraded. 

The  F-22 

The  F-22  processor,  selected  in  1990  as  the  best  available  technology,  is  the 
i960mx,  which  operates  at  8-10  MIPS.  The  electronic  warfare  system  requires  10  such 


127  Source  F-22,  AWACS,  and  Joint  STARS  Program  Offices. 
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processors  to  perform  its  mission.  Today  a  single  processor  could  provide  the  same 
capability.  Because  the  supplier  is  ending  production,  the  Air  Force  is  making  a  final  buy 
of  the  number  required  until  an  upgraded  processor  can  be  incorporated  into  the  design  of 
the  aircraft.  Processor  upgrade  programs  are  scheduled  for  2000  and  2005,  when  the  F- 
22  is  expected  to  reach  operational  capability.128 

Computer  processors  are  not  the  only  area  of  technology  that  is  rapidly  changing. 
Other  areas  include  digital  signal  processors,  memory,  sensors,  communication  systems, 
autonomous  control,  and  navigation  technologies.  Many  of  these  technology  advances 
are  being  driven  by  the  rapid  advances  within  the  commercial  electronics.  Military 
aircraft,  ships,  and  space  systems  rely  heavily  on  such  electronic  systems  to  provide 
communication  and  control. 


D.  Slow  Response  to  New  or  Emerging  Threats 

Emerging  threats  based  on  new  technology  or  a  unique  combination  of  existing 
technologies  can  pose  a  significant  challenge  to  U.S.  forces,  and  leave  them  exposed. 
Closing  the  performance  gap  and  quickly  developing  counter-systems  is  an  important 
aspect  of  maintaining  technologically  superiority. 

One  example  of  the  U.S.  failure  to  pursue  this  strategy  has  been  development  of  a 
counter-system  to  the  Soviet  Archer  AA-1 1  off-boresight  air-to-air  missile.  Off-boresight 
missiles  can  attack  aircraft  at  a  wider  angle  then  standard  air-to-air  missiles  allowing 
aircraft  equipped  with  them  to  fire  on  opponents  at  greater  angles  from  the  nose  of  the 
aircraft  and  significantly  increasing  their  chances  of  killing  their  the  opponent  before 
being  shot  down. 


128  Source  F-22  Program  Element  Monitor  and  F-22  Program  Office. 
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U.S.  Response  to  Off-Boresight  Air-to-Air  Missiles 

The  United  Stated  learned  of  the  Soviet  AA-1 1  Archer  Off-Boresight  Missile  and  its 
Helmet-Mounted  Cueing  system  in  1985,  when  it  photographed  Soviet  fighters  carrying  the 
new  missiles  near  Finland.  In  1989,  after  the  unification  of  Germany,  the  U.S.  obtained 
access  to  many  operational  AA-1 1  missiles  from  the  former  East  German  Air  Force. 
Analysis  and  flight  tests  of  the  AA-missiles  and  the  cueing  system  in  Mig-29s  indicated  a 
“very  significant”  combat  advantage  over  F-16s  in  close  air-to-air  combat. 129  The  Israelis 
minimized  their  exposure  to  the  new  Soviet  missile  by  quickly  developing  and  fielding  a 
more  capable  off-boresight  missile,  which  they  tested  within  three  years  and  had  operational 
within  nine  years.  Both  the  Russian  and  Israeli  missile  systems  are  now  sold  to  a  number  of 
countries.130 

The  U.S.  Air  Force  had  long  seen  the  significant  advantage  provided  by  off- 
boresight  missiles  in  close  air  combat.  Studies  in  1970  of  the  F-15  and  the  proposed 
AIM-82  Off-Boresight  Missile  showed  a  very  dramatic  effect  with  kill  ratios  increasing 
dramatically  over  similarly  non-equipped  aircraft.131  The  AIM-82  missile  was  never 
developed.  In  a  1995  article  in  Aviation  Week,  General  Roland  Yates,  retired  Air  Force 
Materiel  Command  commander,  said,  “The  off-boresight  missile  and  helmet  mounted 
cueing  system  dominates  the  close-in  fight  within  3  miles.  If  an  enemy  has  the  capability 
and  you  do  not,  he’s  going  to  kill  you.  If  he  can  see  you,  you  are  dead.  Even  a  9  g  turn 
will  not  make  any  difference.” 132  A  former  senior  Joint  Chiefs  of  Staff  officer  stated, 

“  For  the  Air  Force  and  Navy  not  to  have  a  helmet-mounted  cueing  sight  and  an  off- 


129  Missile  Handicap,  Part  1 .  U.S.  Intensifies  Efforts  to  Meet  Missile  Thrreat”  Aviation  Week  and  Space 
Technology.  16  October  1995.  Pg.36. 

130  Missile  Handicap,  Part  1 .  U.S.  Intensifies  Efforts  to  Meet  Missile  Threat”  Aviation  Week  and  Space 
Technology.  16  October  1995.  Pg.36. 

131  The  study  was  conducted  by  LTC  Larry  Welch  in  1970  and  was  know  as  the  TAC  Avenger  Study.  It 
showed  very  dramatic  advantages  of  Off-boresight  missiles.  LTC  Welch  later  became  the  Air  Force  Chief 
of  Staff.  It  is  detailed  in  the  Pentagon  Paradox  by  James  Stevenson.  Naval  Institute  Press  Annapolis  MD 
1993. 

132  Missile  Handicap,  Part  1 .  U.S.  Intensifies  Efforts  to  Meet  Missile  Threat”  AWST  16  Oct  1995.  Pg.36. 
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boresight  missile  is  absolutely  criminal.”  133 

The  U.S.  system  developed  in  response  to  the  new  threat  is  the  AIM-9X,  which  is 
expected  to  reach  operational  capability  in  2002, 17  years  after  the  threat  was  identified. 
Currently,  no  U.S.  fighter  has  any  off-boresight  missile  capability. 

E.  Slow  Response  to  Known  Safety  Problems 

The  current  development  process  is  also  often  slow  to  respond  to  identified  safety 
requirements.  Two  high-visibility  programs  that  highlighted  this  slow  response  are  the 
integration  of  Traffic  Collision  and  Avoidance  Systems  and  Global  Positioning  System 
Receivers  on  military  aircraft. 

Traffic  Collision  Avoidance  System 

The  Traffic  Collision  and  Avoidance  System  (TCAS)  was  first  officially 
recommended  for  installation  in  Air  Force  aircraft  in  1994  after  a  near  collision  among 
two  B-52s  on  an  around-the-world  exercise  and  a  Saudi  Arabian  747  over  the 
Mediterranean  Sea.134  Similar  TCAS  equipment  had  been  previously  ordered  installed  on 
all  commercial  transport  aircraft  in  1988.  A  safety  officer  implored  that  the  equipment 
also  be  added  to  military  aircraft  after  another  near-miss  in  1995,  when  a  C-5  flew 
between  a  KC-10  and  the  flight  of  F- 16s  it  was  refueling. 

After  the  crash  of  the  T-43  (Boeing  737)  carrying  Secretary  of  Commerce  Ron 
Brown  in  1996,  the  Air  Force  was  ordered  to  install  the  TCAS  system  on  its  aircraft. 
Installation  on  VIP  transport  aircraft  is  to  be  completed  by  2001,  and  on  larger  transports, 
which  carry  troops  and  cargo,  by  2006.  This  response  came  too  late  for  the  crews  of  a 
U.S.  C-141  and  a  German  C-130,  which  collided  off  the  coast  of  Africa  in  September 
1997,  killing  33  people,  including  9  U.S.  service  members. 


133  Missile  Handicap,  Part  1.  U.S.  Intensifies  Efforts  to  Meet  Missile  Threat”  AWST  16  Oct  1995.  Pg.36. 

134  This  example  is  based  on  the  Article  by  Matthew  Brelis  and  Stephen  Kurkjian.  “  Ill  Equipped  Air  Force 
Plane  Haunts  Widow” ,  Boston  Globe.  December  3  1997.  Page  1. 
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GPS  Receivers  on  Military  Aircraft 

The  long  development,  purchase,  and  installation  time  in  providing  Global 
Positioning  Systems  navigation  equipment  on  military  aircraft  has  produced  similar 
results.  Commercial  GPS  receivers  have  been  widely  available.  FAA  certified  aircraft 
GPS  receivers  are  available  for  commercial  and  private  aircraft  for  less  then  several 
thousand  dollars  for  aircraft  for  many  years.  Portable  handheld  receivers  with  moving 
map  displays  and  aeronautical  databases  designed  for  aircraft  use  are  available  for  less 
than  $600.  The  lack  of  GPS  capability  was  sited  as  one  of  the  contributing  factors  in  the 
1996  crash  of  a  military  Boeing  737  that  killed  Commerce  Secretary  Ron  Brown,  34 
business  leaders,  and  crew  others.  The  pilot  was  off  coarse  on  a  non-precision  approach 
and  collided  with  a  mountain.  It  was  found  that  GPS  could  have  helped  avoid  the 
accident.  Following  the  Brown  accident  GPS  equipment  was  ordered  installed  on  all 
military  aircraft.  GPS  could  have  helped  avoid  other  accidents  as  well  including  the 
crash  of  a  C- 130  presidential  support  plane  in  Wyoming  which  went  off  coarse  at  night 
and  hit  a  mountain.  The  lack  of  GPS  navigation  also  could  have  helped  avoid  an  Army 
helicopter  inadvertently  crossing  the  DMZ  into  North  Korea;  one  of  the  pilots  died  and  a 
international  incident  ensued  when  the  North  Koreans  shot  down  the  helicopter.  GPS 
navigation  has  been  in  development  since  1973  and  has  been  commercially  available 
since  1992.  Integration  of  GPS  systems  into  all  military  aircraft  is  not  expected  to  be 
complete  until  after  2002. 


Appendix  2.  Impact  of  Long  Development  Times 


379 


F.  Effects  of  Development  Time  on  Cost 

Long  development  times  also  impact  the  cost  of  the  systems  that  we  buy.  It  leads 
to  higher  development  cost  and  less  money  being  spent  on  producing  the  products. 

Increased  Cost  of  Development 

Conventional  wisdom  indicates  that  the  longer  the  development  time,  the  more  a 
project  will  cost.  The  Packard  Commission  concurred,  stating  that  “Time  is  money,  and 
experience  argues  that  a  ten-year  acquisition  cycle  is  clearly  more  expensive  than  a  five- 
year  cycle.” 135  There  is  significant  evidence  of  this  effect  in  commercial  development 
efforts.'36  Unfortunately,  no  data  estimating  the  cost  of  different  development  schedules 
are  available  for  specific  military  projects.  The  cost  models  now  used  by  DoD  and  the 
services  do  not  account  for  the  effects  of  time. 

Data  on  major  defense  acquisition  projects  available  from  Rand  show  a  positive 
correlation  between  development  time  and  cost.  Of  the  ACAT I  programs,  those  that  take 
less  than  7  years  to  complete  have  an  average  development  cost  of  $1 .2  billion.  Projects 
that  take  between  7  and  14  years  have  an  average  cost  of  $  1 .8  billion.  Those  taking  over 
14  years  average  $3.6  billion  in  development  cost.137 


135  President’s  Blue  Ribbon  Commission  on  Defense  Management.  “  A  Formula  For  Action:  A  Report  to 
the  President  on  Defense  Acquisition”  (The  Packard  Commission  Report)  Washington  D.C.  April  1986.p. 
8. 

136  See  discussion  in  Chapter  2. 

137  For  the  123  projects  in  the  Rand  database  with  the  necessary  information,  the  Pearson  correlation 
coefficient  between  the  length  of  project  schedules  and  cost  of  development  is  positive  0.25,  with  a  two- 
tailed  significance  level  of  0.005.  This  indicates  that  though  there  is  considerable  scatter  in  the  data,  longer 
programs  on  average  do  cost  more. 
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Figure  A2-1:  Cost  by  Years  in  Development  for  Major  Defense  Acquisition 
Programs  (Data  from  Rand  SAR  Database) 


Similarly,  cost  and  schedule  data  on  154  projects  of  all  sizes  included  in  the 
surveys  conducted  as  part  of  this  research  effort  indicate  that  the  correlation  between 
development  time  and  cost  is  both  positive  and  statistically  significant.138  The  best  fit 
line  through  the  development  cost  and  the  development  time  for  the  projects  surveyed 
indicated  that  the  development  cost  was  best  correlated  with  the  fourth  power  of  the 
development  time. 

4 

Dev  Cost  ($M)  ~  (0.03  x  Dev  Time  (months)  +  1.36) 


138  These  surveys  and  the  responses  from  them  will  be  discussed  at  length  in  Chapter  5-10. 
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Figure  A2-2:  Plot  of  Development  Cost  and  Development  Time  for  154  Projects 
Surveyed  as  Part  of  this  Research.  4th  power  scale  for  cost.  Best  fit  line  drawn 
through  Data. 


This  does  not  necessarily  mean  that  a  project  scheduled  to  take  longer  than 
another  will  necessarily  result  in  dramatic  increases  in  development  costs,  only  that 
across  all  projects,  development  cost  is  associated  with  the  fourth  power  of  the  schedule. 
This  relationship  may  be  complicated  by  a  host  of  factors  other  than  simply  long 
development  times,  including  complex  requirements  for  the  weapons.  Also,  long 
development  times  may  result  in  design  changes  in  response  to  changing  threats, 
technologies,  and  priorities,  raising  costs.  However,  the  data  does  suggest  that  longer 
programs  not  only  cost  more  -  they  cost  a  lot  more. 

More  Funds  Towards  Development  —  Less  Towards  Production 

Not  only  do  longer  programs  cost  more  to  develop  but  a  larger  percentage  of  the 
total  project  cost  is  consumed  during  development.  Data  from  the  Rand  SAR  database 


£  2  2  pq  cq 
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indicate  that  projects  requiring  less  than  14  years  of  development  time  saw  27  percent  of 
their  cost  go  to  development  and  73  percent  go  to  production.  Project  with  development 
times  over  14  years  had  46  percent  of  their  cost  go  to  development  and  54  percent  go  to 
production. 


3 

P 


Development  Time 


Figure  A2-1:  Percentage  of  Program  Cost  Going  to  Development  and  Production, 
by  Development  Time139 


As  development  programs  have  lengthened  over  the  years,  so  has  the  percentage 
of  DoD  funding  going  to  research,  development,  test,  and  evaluation  (RDT&E).  RDT&E 
funding  for  1997  is  $32  billion,  representing  42  percent  of  total  DoD  investment  funding 
(RDT&E  plus  production).  This  is  the  highest  percentage  ever.  The  rising  percentage 
spent  on  RDT&E  means  that  a  smaller  percentage  is  available  for  producing  new  systems 
or  enhancing  the  operations,  training,  and  readiness  of  existing  forces.  While  both 
RDT&E  and  production  funds  have  been  cut  significantly  in  the  last  10  years,  the 
RDT&E  accounts  have  sustained  smaller  decreases.  This  has  resulted  in  fewer  new 
systems  being  fielded  and  made  available  for  the  warfighter. 
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Figure  A2-2:  Percent  of  DoD  Investment  Dollars  (RDT&E  +  Procurement)  Going  to 
Research,  Development,  Testing,  and  Evaluation140 


139  Source  of  Data:  Rand  SAR  Data  Base 

140  Based  on  DoD  Annual  Budget. 
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Appendix  3a 


Organizations  and  Roles 


A.  Organizations  and  Roles 

There  are  many  organizations  involved  with  the  development  of  any  new  weapon  system. 
The  primary  organizations  are  the  Department  of  Defense,  the  individual  services,  and  their 
subordinate  organizations,  the  defense  contractor  community,  and  the  administration  and 
Congress. 


A.l.  The  Department  of  Defense 

The  Department  of  Defense  was  formed  in  1947  to  oversee  and  coordinate  between  the 
previously  independent  services  and  to  provide  a  single  advisor  to  the  President  on  national 
defense  matters.  Over  the  last  30  years  the  Department  of  Defense  (DoD)  has  assumed  a  larger 
role  in  large  development  efforts  through  a  series  of  acquisition  related  reforms.  However,  the 
Department  of  Defense  plays  many  roles,  of  which  acquisition  and  development  is  only  one  part. 

The  Department  of  Defense  is  organized  into  an  operational  side  and  a  support  side.  The 
operational  side  consists  of  the  unified  or  war  fighting  commands.  These  commands  are 
responsible  for  planning  and  executing  wars  and  other  operations.  These  are  the  organizations 
that  put  bullets,  bombs,  and  missiles  on  target  and  carry  out  the  warfighting  responsibilities.  The 
unified  commands  are  supplied  with  fighting  units  from  all  three  services  to  form  a  force.  These 
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organizations  are  headed  by  Commanders  in  Chief  or  CINCs  who  report  directly  to  the  President 
and  the  Secretary  of  Defense  as  the  National  Command  Authorities. 

The  support  side  of  DoD  includes  the  military  services  (Army,  Navy,  Air  Force,  and 
Marines),  the  Joint  Chiefs  of  Staff,  Defense  Agencies,  and  much  of  the  OSD  Staff.  These 
organizations’  primary  purpose  is  to  provide  trained,  combat  ready  forces  and  equipment  to  the 
unified  commanders  to  ensure  they  can  accomplish  their  warfighting  missions.  Other  mission  of 
the  various  organizations  are  to  provide  effective  guidance,  oversight,  and  management  control 
over  Department  of  Defense  agencies,  and  the  services  to  ensure  they  do  so  effectively  and 
within  the  guidance  provided  by  the  President  and  congress. 


Figure  A3a-3:  DoD  Organizational  Chart. 
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A.2.  DoD  Role  in  Product  Development 

One  role  DoD  plays  in  the  development  of  new  weapon  systems  stems  from  its  roles  as 
the  developer  of  national  military  strategy,  its  role  in  the  budget  process  through  guidance  and 
budget  submission,  and  its  role  as  a  regulator  and  overseer  of  the  service  organizations.  DoD  has 
a  direct  role  in  the  authority  for  major  defense  acquisition  programs.  Following  a  series  of 
reforms  the  responsibility  of  DoD  for  acquisition  program  has  increased  significantly  since  the 
1960’s. 

Currently  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology 
(OUSD(A&T))  is  the  primary  office  responsible  for  all  DoD  Acquisition  activities.  The 
OUSD(A&T)  is  the  Defense  Acquisition  Executive  responsible  for  supervising  the  performance 
of  the  entire  DoD  acquisition  system.  He  chairs  the  Defense  Acquisition  Board  and  serves  as  the 
DoD  procurement  executive.  His  organization  publishes  policies  and  procedures  in  the  form  of 
DoD  regulations,  such  as  DoD  5000  series  and  Defense  Federal  Acquisition  Regulations,  which 
govern  the  operations  of  the  DoD  acquisition  system  and  the  administrative  oversight  of  defense 
contractors. 

OUSD  (A&T)  also  plays  a  direct  role  in  the  management  of  major  defense  acquisitions 
programs  as  he  is  the  Acquisition  Executive  responsible  for  the  execution  of  the  acquisition 
program,  he  chairs  the  Defense  Acquisition  Board,  and  he  serves  as  the  source  selection  authority 
for  many  major  programs.  In  these  roles  for  major  programs  managed  by  OSD  he  and  his  staff 
develops  acquisition  plans,  strategies,  guidance,  and  assessments  in  support  of  the  acquisition 
milestone  reviews. 

DoD  also  has  several  development  agencies  that  report  directly  to  it  such  as  the  Ballistic 
Missile  Defense  Organization,  the  Defense  Airborne  Reconnaissance  Office,  the  Advanced 
Research  Project  Agency,  and  the  Defense  Information  Systems  Agency  which  carry  out  their 
own  development  efforts  but  report  to  OUSD  (A&T)  for  acquisition  efforts. 

DoD’s  role  in  the  acquisition  process  on  a  majority  of  non-major  defense  development 
projects  is  primarily  oversight,  and  coordination.  The  planning  and  executing  of  acquisition 
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programs  is  carried  out  primarily  by  the  individual  services.  The  level  of  DoD  involvement 
depends  primarily  on  the  size,  scope,  and  cost  of  the  acquisition  effort. 

Another  DoD  organization  that  is  involved  in  the  development  and  acquisition  of  any 
new  weapon  system  is  the  Director  of  the  Defense  Operational  Test  and  Evaluation.  This 
separate  organization  reports  directly  to  the  Secretary  of  Defense  and  plays  a  significant  role  in 
the  testing  and  evaluation  of  major  defense  acquisition  efforts  to  ensure  that  they  meet  the 
operational  requirements  that  were  established  for  a  development  program  and  to  ensure  that 
systems  are  operationally  effective  prior  to  large  scale  production  and  deployment.  Congress, 
following  a  series  of  operational  problems  with  new  acquisition  programs,  mandated  specific 
testing  and  evaluations  for  any  major  program  before  a  production  decision  can  be  made. 


B.  Services 

The  services’  missions  are  to  organize,  train,  supply,  and  equip  forces  to  ensure  that  they 
are  ready  to  support  the  unified  commands  in  executing  operational  directives.  The  Air  Force, 
Army  and  Navy  are  all  separated  into  operational  and  support  organizations. 141  The  Air  Force’s 
operational  organizations  include  air  wings,  missile  units,  and  space  operations  units.  This  side 
of  the  Air  Force  consists  of  the  Major  Commands  (MAJCOMs)  such  as  Air  Combat  Command, 
Air  Mobility  Command,  and  Air  Force  Space  Command.  The  MAJCOMs  consists  of  many  self 
sufficient  fighting  units  (Wings)  that  are  maintained  in  a  state  of  readiness  to  enable  them  to  be 
used  by  their  various  unified  commands  in  the  event  that  they  are  needed. 

These  operational  units  are  supported  by  the  support  community  to  acquire,  to  maintain, 
and  to  man  their  equipment  as  they  train  and  fight.  The  support  side  is  responsible  to  ensure  that 
they  are  properly  manned,  trained,  and  equipped  with  the  tools  that  will  allow  them  to  be 
successful  in  their  combat  roles. 

Within  the  Air  Force  support  function  there  are  several  different  communities  or 
functional  organizations  that  play  different  roles.  The  primary  support  organizations  include 


141  Many  would  include  the  Marines  and  Special  Operations  Command  as  a  separate  services. 
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logistics,  acquisition,  testing,  personnel,  financial,  and  contracting  organizations.  The  primary 
communities  that  affect  the  development  of  a  new  weapon  system  are  the  planning  and 
requirements  communities,  the  acquisition  community,  the  research  and  development 
community,  and  the  budget  community.  Other  groups  that  are  also  involved  are  the  testing 
community,  the  logistics  community,  and  the  personnel  community.  These  support  organizations 
range  from  the  wing  level  officers,  to  the  major  command  staff,  to  the  Air  Staff  in  the  Pentagon, 
to  the  secretariat  organizations.  Personnel,  logistics  and  contracting  are  part  of  the  Headquarters 
Staff  and  report  to  the  Chief  of  Staff.  The  acquisition  and  financial  communities  are  part  of  the 
secretariat  and  report  to  the  Secretary  of  the  Air  Force.  This  discussion  will  focus  on  those 
organizations  that  most  significantly  impact  the  acquisition  process.  These  communities  are 
discussed  because  each  plays  a  different  role  in  the  acquisition  process  and  are  represented  on 
many  of  the  current  integrated  product  teams  associated  with  the  development  of  a  new  system 
and  the  Air  Force  resource  allocation  process  that  directly  impact  the  acquisition  process. 


USAF 


SAF/AQ  Acquisiton  DCS/XO  Plans  and  Operations  Air  Combat  Command 

SAF/FM  Financial  Management  DCS/SC  Comm  and  Info  Air  Mobility  Command 

SAF/SN  Space  DCS/DP  Personnel  AF  Space  Command 

SAF/IA  International  Affairs  DCS/LG  Logistics  AF  Special  Ops  Command 

SAF/MI  Manpower,  Res,  Inst  &  Env  AF/IN  Intellegence  AF  Materiel  Command 

SAF/SB  Small  Business  Utilization  AF/TE  Test  and  Evaluation  Separate  Reporting  Units 

SAF/GC  General  Council  AF/SE  Safety 

SAF/IG  Inspector  General  AF/CE  Civil  Engineering  Each  consists  of  Wings, 

SAF/LL  Legislative  Liaison  AF/PE  Programs  and  Groups,  and  Squadrons. 

Evaluation 

Judge  Advocate  General 

- -  AF/ANG  Air  National  Guard 

AF/RE  Air  Force  Reserves 


Figure  A3a-4:  Air  Force  Organization  at  the  Secretariat  and  Air  Staff  Levels 


392  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


B.l.  Planning  Community 

Future  planning  in  the  Air  Force  is  currently  done  by  three  groups:  the  new  long  range 
planning  group  HQ  USAF/LP,  the  Mission  Area  Teams,  and  the  operational  commands.  The 
long  range  planning  group  which  was  started  in  1996  is  expected  to  look  at  the  requirements  and 
force  structure  25  to  40  years  in  the  future.  It  is  centered  in  the  Pentagon  as  part  of  the  Air  Staff. 

The  Mission  Area  Teams  and  their  the  associated  Technical  Planning  Integrated  Product 
Teams  (TPIPTs)  develop  mission  area  plans  that  look  out  to  the  future  25  years.  The  Mission 
Area  Plans  are  divided  into  mission  categories  (such  as  aerospace  control  which  covers  offensive 
counter-air,  defensive  counter-air  and  theater  missile  defense)  and  are  designed  to  provide  long 
range  direction  to  research  and  development  investments  by  the  scientific,  development,  and 
contractor  communities.  The  major  commands  lead  the  Mission  Area  Teams  and  the  TPIPTs 
are  managed  by  the  Air  Force  Materiel  Command  and  their  associated  product  centers. 

Planning  for  5  years  is  done  on  an  ad  hoc  basis  at  the  operational  command  level.  At  the 
operational  commands  the  organizations  that  are  involved  with  planning  are  Command  XP 
organizations. 

The  processes  used  by  this  community  are  defined  by  the  regulations  below. 

CJCS  7  Joint  Strategic  Planning  System 

AFPD  10-14  Modernization  Planning 

AFI  10-1401  Modernization  Planning  Documentation 

The  planning  community  is  typically  staffed  by  operational  military  officers  with 
experience  in  the  various  weapon  systems.  A  planning  position  at  the  major  command  or  the 
Pentagon  is  considered  a  significant  staff  position  which  is  part  of  a  mix  of  experiences  expected 
to  make  higher  ranks.  Operational  officers’  paths  for  advancement  are  through  the  operational 
side  of  the  Air  Force  becoming  group  and  wing  commanders.  Staff  officers  typically  spend 
between  two  and  four  years  in  these  positions  before  returning  to  their  operational  units. 
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B.2.  Requirements  Community 

The  requirements  community  is  responsible  for  determining  what  new  systems  are 
needed  and  what  the  capabilities  of  a  new  system  must  be  to  undertake  and  succeed  at  the 
missions  assigned  to  Air  Force  units.  There  is  significant  overlap  between  the  requirements  and 
the  planning  communities.  The  requirements  community  consists  primarily  of  two  different 
levels  of  organizations,  one  at  the  major  command  headquarters,  and  one  at  the  Pentagon.  The 
Deputy  for  Requirements  at  most  operational  commands  organizations  are  responsible  to 
evaluate  the  current  capabilities  and  identify  deficiencies  between  the  current  capabilities  and  the 
required  capabilities  to  carry  out  the  existing  assigned  operational  mission  or  those  that  are 
projected  to  exist  in  the  future.  They  document  these  deficiencies  and  develop  several 
documents  including  the  Mission  Needs  Statements  which  are  precursors  to  the  acquisition 
process  and  the  Operation  Requirements  Documents  (ORDs)  which  specify  the  capabilities  and 
the  operational  concept  of  any  new  or  modified  system.  The  requirement  community  is  headed 
by  HQ  USAF/XOR  who  reviews,  staffs,  and  maintains  the  requirements  documents.  Depending 
on  the  size  and  scope  of  the  project,  the  operational  requirements  must  be  validated  by  either  the 
Joint  Operational  Requirements  Council  (JROC)  at  the  Vice  Joint  Chief  of  Staff  level,  its  Air 
Force’s  Equivalent  AFROC,  or  the  Air  Force  Chief  of  Staff. 

Each  major  development  program  and  program  area  is  represented  by  an  requirements 
officer  in  HQ  USAF/XOR  who  staffs  and  defends  the  requirements  documents  through  the 
approval  process. 

The  processes  used  by  this  community  are  defined  by  the  regulations  below. 

DoD  5000. 1  Defense  Acquisition 

DoD  5000.2  Defense  Acquisition  management  Documentation  and  Procedures 

CJCS  3 1 70  Requirements  Generation  System 

CJCS  MCM  76-92  Charter  for  the  Joint  Operational  Requirements  Council 

AFPD  10-6  Mission  needs  and  Operational  Requirements 

AFI 1 0-60 1  Mission  Needs  and  Operational  Requirements  Guidance  and  Procedures 

AFI 33-103  C4I  systems  requirements ,  Development  and  Processing 
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The  requirements  community  is  typically  staffed  by  operational  military  officers  with 
experience  in  the  various  weapon  systems.  A  requirements  position  at  the  major  command  or  the 
Pentagon  is  considered  a  desirable  staff  position.  Staff  officers  typically  spend  between  two  and 
four  years  in  these  positions  before  returning  to  their  operational  units.  These  officers  typically 
have  not  worked  in  the  acquisition  community  prior  to  their  assignments  as  requirements 
officers.  The  path  for  advancement  for  these  officers  is  through  their  operational  commands. 


B.3  Programming  Community 

The  programming  community  is  responsible  for  developing,  integrating  and  maintaining 
the  Future  Year  Defense  Plan  and  the  program  objective  memorandum  which  authorizes  the 
services  to  carry  out  specific  functions.  The  programming  community  is  centered  in  HQ 
USAF/PE  and  interacts  with  representation  from  every  organization.  The  role  of  programming 
is  similar  to  the  authorizing  committees  in  congress  which  allows  activities  to  undertaken  by  the 
government.  The  programming  function  is  a  central  piece  of  the  Planning,  Programming  an 
Budgeting  System  (PPBS)  that  is  used  to  plan  and  budget  all  Air  Force  activities. 

The  programming  community  organizes  the  Air  Force  corporate  deliberative  process 
using  with  an  array  of  integrated  product  teams  and  panels  at  various  levels  to  review  and 
provide  a  wide  array  of  options  for  the  development  of  the  Air  Force  program  and  requested 
budget.  The  lowest  level  in  the  programming  community  is  the  Program  Element  Monitor  who 
is  the  official  representative  of  that  activity  within  the  Pentagon. 

The  Air  Force  Planning  community  has  its  own  set  of  regulations  that  guide  its  actions. 
They  are: 


DoDD  7045.14 
AFPD  16-5 
AFI 16-501 
AFPD  16-3 
AFI  16-301 


Planning,  Programming,  and  Budgeting  System 
Planning,  Programming,  and  Budgeting  System 
Control  and  Documentation  of  Air  Force  Programs 
Priorities  for  Resources  Management 
US  Air  Force  Priority  System  for  Resources  Management 
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The  programming  community  is  also  typically  staffed  by  mid  grade  operational  military 
officers  with  experience  in  the  various  weapon  systems  and  augmented  with  defense  civil 
servants.  An  air  staff  programming  position  at  the  Pentagon  is  considered  a  significant  staff 
position.  Staff  officers  typically  spend  between  two  and  four  years  in  these  positions  before 
returning  to  their  operational  units  and  major  commands. 


B.4.  Budget  Community 


The  budget  community  consists  of  the  Secretary  of  the  Air  Force  Financial  Management 
Office  (SAF/FM)  and  the  FM  organizations  at  each  base  and  the  product  centers.  SAF/FM  is 
responsible  for  building,  submitting,  and  distributing  the  Air  Force  budget.  They  produce  the 
Budget  Estimate  Submission  (BES)  based  on  the  FYDP  programming.  The  BES  goes  to  DoD 
to  become  part  of  the  Defense  portion  of  the  President’s  Budget.  Once  the  budget  is  passed  and 
signed,  the  budget  community  is  responsible  for  distributing  the  budget  as  dictated  by  Congress 
and  tracking  expenditures  to  ensure  that  no  money  is  spent  that  is  not  allocated  and  appropriated 
for  a  specific  project  element.  Allocated  funding  is  usable  for  1, 2,  3,  or  6  years  numbers  of  year 
depending  upon  which  designation  is  it  assigned.  FM  tracks  the  authorized  moneys  and  ensures 
that  no  funds  are  spent  that  are  not  current  and  authorized  by  law. 

The  Budget  Community  has  specific  regulations  and  laws  that  govern  direct  their  actions. 
The  instructions  detail  the  procedures  used  to  develop  the  budget  and  distribute  the  money. 
Below  are  the  regulations  that  effect  the  product  development  process  and  the  development  of 
the  budget  submission. 


AFPD65-1 

Management  of  Financial  Services 

AFPD  65-2 

Internal  Management  Controls 

AFPD  65-5 

Cost  and  Economics 

AFPD  65-6 

Budget 

AFI 65-201 

Internal  Management  Controls  Program 

AFI  65-501 

Economic  Analysis 

AFI  65-502 

Inflation 

AFI  65-503 

US  Air  Force  Cost  and  Planning  Factors 

AFI  65-504 

Independent  Review  of  Commercial  Activity  Cost  Comparisons 

AFI  65-601V1 

Budget  Guidance  and  Procedures 

AFI  65-601V2 

Budget  Management  for  Operations 

AFI  65-603 

Official  Representation  Funds-Guidance  and  Procedures 
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SAF/FM  is  a  staffed  by  a  mixture  of  military  officers  and  defense  civilians.  Military 
officers  in  the  budget  field  are  known  as  financial  analysts  and  they  receive  specialized  training. 
While  there  is  significant  overlap  and  cross  training  with  the  acquisition  officers,  the  career 
tracks  are  different  and  they  have  separate  routes  for  advancement.  Many  officers  assigned  as 
financial  analysts  spend  a  majority  of  their  careers  in  various  part  of  the  budget  field,  but  spend  2 
to  4  years  in  each  position.  Defense  civil  servants  can  spend  tens  of  years  in  a  single 
organization.  Many  financial  analysts  have  had  assignments  in  program  offices  and  have 
significant  experiences  with  the  acquisition  process.  Financial  Analysts  will  become  base, 
center,  and  Air  Force  comptrollers  as  they  advance  their  careers. 


B.5.  Acquisition  Community 

The  Acquisition  Community  consists  of  the  Secretary  of  the  Air  Force  for  Acquisition, 
the  Program  Executive  Officers,  Air  Force  Materiel  Command,  the  Product  and  Logistics 
Centers,  and  the  Program  Offices.  These  organizations  execute  the  acquisition  effort  and  buy 
the  systems  the  Air  Force  needs. 


Assistant  Secretary  for  Acquisition 

The  SAF/AQ  organization  is  the  Air  Force’s  Pentagon  focal  point  for  acquisition 
efforts.  The  Secretary  of  the  Air  Force  for  Acquisition  is  also  the  Senior  Acquisition  Executive 
(also  known  as  the  Component  Acquisition  Executive  -  CAE)  and  is  responsible  for  any  service- 
run  acquisition  effort.  The  Senior  Acquisition  Executive  reports  to  both  the  Secretary  of  the  Air 
Force  and  the  Under  Secretary  of  Defense  for  Acquisition  and  Technology  on  acquisition  efforts. 
SAF/AQ  is  divided  into  the  different  mission  areas  that  manage  related  projects  and  development 
efforts  -  Global  Power  (combat  aircraft),  Global  Reach  (airlift  and  tanker  aircraft),  Information 
Dominance,  Space,  and  Weapons.  These  different  areas  are  led  by  Mission  Area  Directors  who 
coordinate  between  different  acquisition  efforts  within  the  same  category  of  weapon  systems. 
Within  these  mission  areas  reside  the  Program  Element  Monitors  (PEMs)  and  Air  Staff  action 
officers  for  each  acquisition  program.  For  modifications  to  operational  systems  for  which 
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responsibility  has  been  transferred  the  operational  community  the  PEMs  are  typically  found  in 
the  XO  community  but  there  is  typically  an  AQ  action  officer  who  oversees  the  project  from 
SAF/AQ.  PEMs  play  a  central  role  in  the  PPBS  system. 


Program  Executive  Officers 

The  Program  Executive  Officers  were  developed  by  the  Packard  Commission  to 
provide  a  direct  line  of  authority  from  the  Senior  Acquisition  Executive  to  the  Program  Manager 
for  the  execution  of  major  defense  acquisition  efforts.  PEOs  have  authority  on  the  execution 
phase  of  the  development  effort.  They  are  organized  around  mission  areas  such  as  Battle 
Management;  Command  and  Control;  Space;  Airlift;  and  Tankers,  Fighters  and  Bombers,  and 
Weapons  and  they  oversee  all  major  defense  acquisition  programs  in  their  respective  areas. 
PEOs  are  supported  by  a  small  staff  of  officers  and  are  located  at  the  Pentagon. 


Air  Force  Materiel  Command 

Air  Force  Materiel  Command  provides  the  support  structure,  people,  and  training 
to  the  program  offices.  It  also  provides  policy  and  guidance  and  functional  expertise  for  the 
program  offices.  Air  Force  Materiel  Command  is  responsible  for  both  the  Product  and  Logistics 
centers  that  develop  and  maintain  the  Air  Force  equipment.  Materiel  Command  with  its  product 
centers  also  leads  the  Technology  Planning  Integrated  Product  Teams  and  does  early  planning 
for  new  weapon  systems  prior  to  the  establishment  of  a  program  office 


Product  Centers 

The  four  product  centers  are  organized  around  different  mission  areas  -- 
Electronic  Systems  Center  focuses  on  electronic  systems;  Aeronautical  Systems  Center  focuses 
on  aircraft  and  weapons  systems;  Space  and  Missile  Center  focuses  on  space  and  missile  related 
programs;  and  Human  Systems  Center  focuses  on  the  personal  equipment  and  human  interface  of 
systems.  Logistics  Centers  are  organized  around  the  operational  systems  that  they  support.  The 
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Commander  of  these  centers  are  designated  Defense  Acquisition  Commanders  (DACs)  and  have 
significant  authority  and  responsibility  for  the  execution  of  non-major  defense  acquisition  efforts. 
Each  development  centers  supports  a  wide  range  of  program  offices  with  people,  facilities, 
training,  and  staff  support. 


Program  Offices 

The  Systems  Program  Office  (SPO)  is  the  acquisition  executing  organization. 
They  actually  put  the  development  effort  on  contract  and  buy  the  systems.  They  are  the  ones 
interfacing  with  the  contractors  and  the  users  and  making  the  day-to-day  program  decisions. 

Each  SPO  is  headed  by  a  System  Program  Director  who  reports  to  either  the  product  or  logistics 
center  commander  or  to  the  program  executive  officer  for  that  program,  depending  on  the  size 
and  cost  of  the  system  under  development.  The  systems  program  office  can  be  dedicated  to  a 
single  product  such  as  the  F-22  or  could  be  a  conglomeration  of  like  products  in  what  is  known 
as  a  basket  SPO.  SPOs  range  in  size  from  small  teams  to  over  five  hundred  personnel. 

These  program  office  carry  out  the  majority  of  the  acquisition  related  activities  to  comply 
with  the  various  laws,  federal,  defense,  and  Air  Force  regulations,  and  program  decision 
guidance  to  met  the  Air  Forces  needs.  Program  Offices  are  not  allowed  to  be  advocates  for  the 
specific  programs  and  their  funding  that  they  are  acquiring  but  instead  are  an  executing  agency 
for  the  operating  commands.  The  operating  commands  and  the  Secretariat  are  the  program 
advocates  and  funding  advocates. 

The  acquisition  community  and  its  actions  are  directed  by  the  following  regulations: 

DoD  5000.1  Defense  Acquisition 

DoD  5000.2R  Defense  Acquisition  Procedures 

AFPD63-1  Acquisition  System 

AFPD  63-3  Competition  Advocacy 

AFI 63-101  Acquisition  System 

AFI  63-107  Integrated  Weapon  System  Management  Program  Planning  and  Assessment 
AFI  63-114  Rapid  Response  Process 
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The  acquisition  community  is  staffed  by  a  mixture  of  military  and  defense  civilians. 
These  officers  have  specialized  training  in  the  acquisition  process.  The  acquisition  officers 
include  program  managers,  developmental  engineers,  and  scientific  career  fields.  Limited 
opportunities  for  cross  training  allows  some  officers  to  obtain  operational  experience.  These 
officers  typically  spend  most  of  their  careers  in  the  acquisition  related  assignments.  They 
typically  spend  between  2-3  years  in  a  specific  position.  Their  path  for  advancement  is  through 
the  acquisition  career  field  to  eventually  become  a  program  director  or  mission  area  director. 


B.6.  Other  Communities  Involved  in  the  Development  Process 
Contracting  Community 

The  contracting  community  is  the  organization  that  maintains  the  regulations  and  ensures 
that  all  Air  Force  contracting  activities  within  the  appropriate  federal  regulations  and  laws.  The 
contracting  community  is  headed  by  the  Deputy  Assistant  Secretary  for  Contracting  under  the 
Assistant  Secretary  for  Acquisition.  The  contracting  community  is  involved  in  every  are  on  every 
base,  in  every  procurement,  and  every  development  program  in  the  Air  Force.  The  contracting 
officer  with  a  warrant  is  the  only  one  who  can  commit  the  Government  to  spend  money.  The 
contracting  officer  is  liable  to  ensure  appropriate  rules  and  laws  are  followed.  In  order  to  obtain  a 
warrant  contracting  officers  attend  significant  training  courses  and  schooling  and  have  specific 
tenure  and  experience  requirements. 

The  contracting  community  supports  all  Air  Force  activities  including  acquisition  efforts. 
A  significant  portion  of  the  contracting  officers  are  involved  in  the  acquisition  process.  At  least 
one  contracting  officer  is  assigned  to  each  program  office.  Contracting  officers  can  be  either  a 
defense  civilian  or  a  military  officer.  Contracting  officers  follow  a  separate  career  track  from 
other  officers  and  defense  civilians.  This  career  path  leads  them  to  become  the  head  of 
contracting  at  the  base,  the  MAJCOM,  and  then  at  the  Pentagon. 
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Science  and  Technology  Community 

The  science  and  technology  community  consists  of  the  labs  and  other  research 
organizations  that  develop  many  of  the  new  technologies  used  in  new  weapon  systems.  The 
Laboratories  are  represented  on  the  Technology  Planning  IPTs  (TPITS)  that  help  define  future 
technology  requirements  for  new  weapon  systems. 


Logistics  Community 

The  logistic  community  is  responsible  for  maintaining  all  systems  in  the  Air  Force 
inventory.  The  Logistics  Community  consists  of  the  Deputy  Chief  of  Staff  for  Logistics  and  the 
Air  Force  Materiel  Command’s  Air  Logistics  Centers  and  Depots,  and  base  logistics 
organizations.  The  logistics  community  is  involved  in  everything  from  toilet  paper  and  pencils 
to  the  most  advanced  weapon  systems.  These  organizations  make  plans  to  support  the  new 
systems.  Any  new  system  that  is  being  developed  and  produced  will  require  support  and 
potential  modification.  Once  a  system  is  fully  operational  authority  for  the  system  is  transferred 
to  the  logistics  community  although  many  systems  that  have  been  operational  for  long  periods 
have  not  been  transferred  such  as  the  AW  ACS  aircraft  system.  The  merger  of  the  Logistics 
Command  and  Systems  Command  and  under  the  Integrated  Weapon  Systems  Management 
(IWSM)  process  this  transfer  is  supposed  to  be  seamless  with  both  the  logistics  and  acquisition 
organizations  supporting  a  weapon  system  reporting  to  a  single  program  director.  The  logistics 
community  is  responsible  for  any  system  upgrades  to  operational  systems  for  which 
responsibility  has  been  transferred. 

The  logistic  community  is  governed  by  the  it  own  set  of  regulations.  Some  that  effect  the 
development  of  new  systems  include: 

AFPD20-1  Logistics  Strategic  Planning  * 

AFI 20-101  Logistics  Strategic  Planning  Procedures 

The  logistics  community  is  staffed  by  a  mixture  of  military  officers  and  defense  civilians. 
These  officers  are  specially  trained  in  the  logistics  career  field  and  have  a  separate  career  track. 
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Despite  the  merger  of  Logistics  and  Systems  command  to  form  Air  Force  Materiel  Command  the 
logistics  and  acquisition  communities  continue  to  be  separate  organizations.  Crossflow  between 
the  two  communities  has  increased.  Officers  still  have  different  career  field  and  career  tracks. 
The  logistics  officer’s  career  track  objective  is  to  become  a  logistics  center  or  depot  commander 
and  then  more  to  the  logistics  staff  at  the  air  staff. 


Personnel  and  Manpower  Communities 

The  personnel  and  manpower  communities  are  involved  supplying  the  people  and  the 
training  required  to  carry  out  the  missions  assigned  to  each  organization.  This  community  serves 
the  entire  Air  Force  and  extends  from  the  Pentagon  to  each  base.  They  are  responsible  for 
authorizing  and  assigning  people  to  each  organization. 

The  Personnel  and  Manpower  organizations  face  several  constraints,  the  first  is  that  the 
total  number  of  people  in  each  service  is  dictated  by  Congress  through  authorized  end  strength 
levels  and  personnel  payroll  accounts.  A  second  factor  that  is  not  common  in  civilian 
companies  is  that  new  people  can  not  be  hired  as  they  are  in  a  civilian  company.  Military  career 
paths  and  promotion  systems  does  not  lend  itself  to  staffing  organizations  from  external  sources 
by  new  hiring.  Military  officers  and  enlisted  people  enter  in  the  junior  ranks  and  progress 
through  the  ranks  at  regular  intervals,  gaining  the  experiences  that  will  enable  them  to  do  the 
higher  level  jobs.  They  also  expect  to  be  able  to  remain  in  the  military  until  they  reach  retirement 
age  at  20  years. 

In  order  to  maintain  the  proper  skill  levels  and  distribution  of  ranks  within  the 
organizations,  great  care  is  taken  to  manage  the  force.  This  is  further  complicated  by  the  large 
number  of  career  fields  that  represents  people  with  certain  experiences  and  a  different  expected 
career  path  which  they  must  be  able  to  follow  in  order  to  have  a  high  probability  of  being 
promoted  and  remaining  in  the  service. 

Because  of  these  constraints,  quickly  changing  the  manpower  levels  and  personnel 
requirements  is  extremely  difficult  if  not  impossible.  If  the  personnel  system  does  not  take  care 
of  the  people  and  their  careers  in  the  services,  many  will  decide  to  exit  the  service.  Again,  these 
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people  can  not  be  replaced  easily  as  it  takes  time  for  people  to  move  up  the  ranks  and  gain  the 
necessary  experiences. 

The  development  of  a  new  weapon  system  will  require  personnel  resources  to  train, 
operate,  and  maintain.  These  manpower  requirements  along  with  the  personnel  required  to  fill 
them  must  be  taken  into  account  in  the  planning  stages  of  any  new  system.  This  is  to  ensure  that 
the  people  with  the  appropriate  skills  are  available  to  accomplish  the  jobs  required. 


The  personnel  and  Manpower  systems  are  governed  by  the  following  regulations: 


AFPD  36-21 
AFPD  36-26 
AFPD  36-5 
AFPD  38-1 
AFI 38-101 
AFI  38-201 
AFI  38-204 
AFI  38-205 


Utilization  and  Classification  of  Air  Force  Military  Personnel 

Military  Force  Management 

Civilian  Personnel  Resource  Management 

Organization 

Air  Force  Organization 

Determining  Manpower  Requirements 

Programming  USAF  Manpower 

Managing  Wartime  and  Contingency  Manpower 


The  personnel  and  manpower  communities  are  manned  by  a  mixture  of  military  officers 
and  defense  civil  servants.  The  career  paths  of  these  officers  is  separate  from  other  officers. 

They  start  at  the  base  level  and  progress  to  staffs  at  various  organizations  leading  to  a  director  of 
personnel  or  Deputy  for  Personnel  position  at  the  Pentagon.  Manpower  officers  spend  3-4  years 
per  assignment.  Civil  servants  can  spend  tens  of  years  in  a  position. 

Testing  Community 

The  testing  community  is  a  semi-autonomous  community  that  reports  to  the  Chief  of 
Staff  and  the  Director  of  Operational  Test  and  Evaluation  at  the  DoD  level.  Prior  to  any  system 
entering  full  production  is  it  required  by  law  to  be  tested  to  ensure  that  it  meets  all  its  operational 
requirements  and  that  it  is  useful  to  the  people  who  need  to  use  it.  The  testing  community  was 
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made  an  independent  organization  responsible  to  the  Secretary  of  Defense  following  several 
system  that  did  not  perform  as  expected. 

The  testing  community  in  the  Air  Force  is  primarily  located  at  the  test  centers  at 
Edwards  Air  Force  Base  and  Eglin  Air  Force  Base.  Planning  for  test  resources  occurs  early  in  a 
program.  Limited  testing  resources  require  planning  to  ensure  the  availability  of  people  and 
equipment  to  undertake  the  testing  programs  for  new  systems. 

The  testing  community  is  governed  by  it  separate  regulations  which  are  listed  below. 


AFPD  99-1 

Test  and  Evaluation  Process 

AFI  99-101 

Developmental  Test  and  Evaluation 

AFI  99-102 

Operational  Test  and  Evaluation 

AFI  99-103 

Test  and  Evaluation  Process) 

AFI  99-105 

Live  Fire  Test  and  Evaluation 

AFI  99-106 

Joint  Test  and  Evaluation) 

AFI  99-109 

Test  Resource  Planning 

The  Operational  Test  Community  is  manned  again  by  a  mixture  of  military  officers, 
defense  civilians,  and  support  contractors.  Military  officers  in  the  testing  community  have  a 
different  career  field  than  the  acquisition  officer  and  the  operational  officers.  Military  officers 
cross  train  into  the  testing  career  fields  by  attending  Flight  Test  School  as  either  a  pilot  or  an 
engineer.  While  there  is  cross  flow  into  and  out  of  the  testing  community  the  testing  community 
is  a  separate  organization  with  a  separate  career  path  that  advances  up  to  a  Test  Center 
Commander. 


C.  External  Organizations 

C.l.  Administration 

The  administration  plays  a  high  level  role  in  setting  national  priorities  and  allocating  the 
federal  budget  between  major  areas.  The  administration  typically  gets  involved  on  only  the 
largest  defense  programs  as  the  time  devoted  to  defense  procurement  issues  is  limited  by  the 
large  number  of  other  issues  addressed  at  that  level.  The  office  of  Management  and  Budget  is 
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the  overseer  of  the  Presidential  Budget  process.  It  implements  the  allocations  of  the  budget 
between  the  various  agencies  and  departments  as  the  president  has  decided. 


C.2.  Congress 

Congress  plays  a  key  role  in  the  development  of  major  weapon  systems  as  they  are  the 
final  authority  on  the  budget  process.  Their  are  two  sets  of  full  committees  in  each  of  the  House 
and  Senate  that  are  involve  in  the  budget  process.  The  authorizing  committees  determine  the 
approved  activities  for  the  services  to  carry  out  and  the  appropriating  committees  allocate  the 
funds  to  carry  out  those  activities.  The  appropriating  and  authorizing  actions  are  addressed  in  the 
discussion  of  the  budget  process. 


C.3.  Defense  Industry 

The  defense  industry  plays  a  central  role  in  the  development  of  any  new  defense  system. 
Nearly  all  development  programs  are  accomplished  by  the  industry  as  the  DoD  does  not  develop 
new  system  in  house.  The  market  this  industry  faces  is  a  near  monopsony  with  the  Department 
of  Defense  the  sole  buyer.  Export  markets  due  exist  but  traditionally  account  for  only  15%  of  the 
total  market.  Further  more  exports  must  be  approved  by  DoD  and  the  state  department. 

The  market  is  very  competitive  of  the  development  of  a  major  system.  Several  large 
prime  contractors  compete  in  most  areas.  The  recent  consolidations  in  the  industry  may  reduce 
the  number  of  competitors  for  certain  weapon  systems,  competition  is  still  intense. 

The  companies  in  the  Defense  market  typically  focus  a  large  percentage  of  their 
companies  on  the  defense  market.  Many  mixed  companies  have  recently  sold  their  defense 
related  components  to  companies  hat  are  focusing  on  the  defense  market.  Due  to  regulatory  and 
oversight  issues  the  defense  units  of  mixed  firms  have  been  separated  from  their  commercial 
counterparts.  These  defense  related  units  focus  on  the  defense  market. 

The  nature  of  the  competition  is  also  significantly  different  than  in  the  commercial  sector. 
Although  it  may  be  done  in  several  stages,  the  competition  for  a  development  contract  for  a 
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major  weapon  system  is  winner  take  all.  The  loser  can  not  typically  re-enter  in  the  future  for  any 
market  share.  The  winner  of  the  development  contract  can  expect  a  long  production  contract, 
many  modernization  efforts,  and  a  long-term  sustainment  program  lasting  upwards  of  40  years  if 
the  system  is  competed  and  enters  development. 

The  contractor  selection  process  is  rigid  and  governed  by  many  laws  and  regulations. 

The  history  of  the  industry  has  exhibited  little  price  sensitivity  with  the  selection  process  being 
focused  on  performance  and  contractor  responsiveness  to  the  details  in  government  request  for 
proposal. 


D.  Organization  Summary 

The  organizations  described  all  play  different  roles  in  the  large  system  that  develops  new 
defense  systems.  The  process  is  separated  into  several  parts  with  different  players  taking  the  lead 
role  at  different  times.  The  center  of  activity  and  responsibility  for  the  development  effort  is 
passed  from  the  operational  commands  to  the  acquisition  community  to  the  defense  industry. 

The  pre-acquisition  activities  are  lead  by  the  using  commands  and  the  programming  and 
budgeting  organizations.  The  acquisition  phase  is  lead  by  the  program  office.  The  actual 
development  phase  is  lead  by  the  contractor  community  as  they  develop,  design,  and  produce  the 
system.  All  of  these  organizations  impact  the  development  of  a  new  defense  system  and  the  time 
that  it  takes  to  produce  and  field. 


Appendix  3b 


Air  Force  Modernization  and  Planning 
Process 


A.  Air  Force  Modernization  Planning  Process 

The  Air  Force  Planning  Process  is  process  to  determine  Air  Force  needs  for  new  or 
improved  capabilities  to  ensure  that  Air  Force  units  can  accomplish  their  assigned  missions. 

The  planning  process  starts  with  the  President’s  National  Security  Strategy  which  in  turn  is 
developed  into  the  National  Military  Strategy  142  and  the  preliminary  Defense  Planning 
Guidance'43  by  OSD  and  the  Joint  Chiefs  of  Staff.  These  provide  direction  to  the  Unified 
Command  CINCs  and  the  military  services  in  terms  of  what  is  required  and  what  resources  will 
be  available  to  carry  out  the  requirements.  The  National  Military  Strategy  is  then,  through  a 
process  called  strategy-to-task,  developed  into  theater,  campaign  and  operational  level  objectives 
and  tasks  that  are  required  to  carry  out  the  military  strategy.  This  airpower  force  planning  is 
accomplished  to  determine  the  Air  Force  needs  and  to  provide  the  Secretary  and  the  Chief  of 
Staff  information  and  assistance  shaping  the  National  Military  Strategy  and  assist  in  the 
development  of  the  Defense  Planning  Guidance.  Airpower  Force  Planning  is  also  accomplished 
to  communicate  with  the  major  commands  in  terms  of  fiscal  constraints  on  force  structure  and 


142  National  Military  Strategy  defines  at  the  highest  level  how  the  US  Military  would  carry  out  it  mission.  It  defines 
which  Unified  Commands  would  undertake  which  missions  and  the  range  of  conflicts  that  the  Military  should  be 
prepared  to  undertake.  The  CINC  then  develop  plans  to  undertake  the  assigned  missions. 
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objective  assessments  of  force  capabilities.  It  also  transfers  information  about  “the  complex 
linkages  among  planning  priorities,  fiscal  reality,  and  potential  Air  Force  programs.” 144 

The  analysis  of  the  capabilities  in  each  Air  Force  mission  areas  are  accomplished  by 
Mission  Area  Teams  (MATs).  Analysis  of  other  cross  mission  Air  Force  activities  is 
accomplished  by  Functional  Area  Teams  (FATs).  See  tables  on  next  page  for  a  list  of  teams. 
These  teams  report  to  the  respective  major  commands  and  are  made  of  primarily  officers  in  the 
requirements  and  operational  specialties.  These  teams  are  led  by  the  headquarters  of  the  major 
commands  or  functional  organizations  and  are  typically  headed  by  a  colonel.  These  Mission 
Area  Teams  are  supported  by  both  subteams  associated  with  each  individual  weapon  system  and 
by  Technical  Planning  Integrated  Product  Teams.  The  sub-teams  are  specialists  on  a  particular 
weapon  system  such  as  the  F-15C.  The  weapon  system  teams  are  headed  by  a  major  or  lieutenant 
colonel  from  the  Major  Command. 

The  Technical  Planning  Integrated  Product  Team  is  contains  the  representation  from  the 
program  offices,  the  laboratories,  and  the  industry.  The  TPIPTs  are  designed  to  identify  the 
potential  future  systems  and  concepts  to  meet  the  Mission  Area  Teams  needs.  They  also  identify 
future  technology  needs  to  direct  laboratory  research.  The  TPIPTs  are  lead  by  the  appropriate 
product  center.  They  also  include  one  or  two  representative  from  industry  through  an  industry 
association.  The  TPIPTs  are  typically  headed  by  a  lieutenant  colonel  from  the  planning  shops  at 
the  product  centers.  The  figure  below  shows  the  relationships  between  the  Mission  Area  Teams, 
the  Technical  Planning  Integrated  Product  Team  and  the  national  and  defense  laboratories.  The 
subsequent  charts  show  the  notional  representation  of  the  Mission  Area  Teams  and  the  weapon 
systems  sub-teams,  the  list  of  mission  area  teams,  and  for  illustration  purposes  the  membership 
of  the  Aerospace  Control  MAT.  Also  shown  are  the  distribution  of  TPIPT  among  the  product 
center  and  the  make  of  the  Aerospace  Control  TPIPT 


143  The  Defense  Planning  Guidance  provides  the  expected  funds  that  each  service  be  authorized  in  the  upcoming 
budget  cycle  and  represents  the  total  dollar  amount  that  each  service  is  expected  to  plan  its  program  to  meet.  It  the 
first  documents  in  the  annual  budget  process. 

144  AFI  10-601 
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MAJCOM  MATS 


Feedback 


Figure  5  Relationship  between  Mission  Area  Teams,  Technical  Planning  Integrated 
Product  Teams  and  Laboratories143. 


145  Modified  from  AQ/XO  briefing  on  Modernization  Planning 
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NOTIONAL  MISSION  AREA  IPT  MEMBERSHIP  AND  WEAPON 
SYSTEM/CAPABILITY/FUNCTIONAL  IPT  MEMBERSHIP 


Core  Members 

Team  Leader  (Normally  0-6) 

TPIPT  Chief 


Mission  Area  Team  Membership  (NOTIONAL) 

Advisors 
SG 
Legal 


InteT 

Logistics 

Functional  Team  Leaders 


Financial 

Contractors 

Experts/Specialists 


Weapons  System/  Capability/ 

Functional  IPT  Membership  (NOTIONAL) 


Core  Members 

Advisors 

Team  Leader  (0-4/0-5) 

*C4 

Program  Management 

•Training 

Functional  Experts 

••Specialists 

-  Operations 

*SG 

-  Logistics 

•Intel 

-  Test  &  Evaluation 

Analysts 

Laboratory(s) 

*CE 

Tactics  Development 
(For  Weapons  Systems) 

PPBS/Financlal 

•R&M 

TPIPT  Representative 

Contractors) 

1  Executive  Committee  ZM 

•SP 

Plans 

Legal 

•Weather 

*  Core  when  serving  as  functional  experts 
**  Same  as  above  but  represent  disciplines  like  avionics 
software  engineering,  bloenvironmental,  etc. 


Table  20  -  Notional  Membership  on  a  Mission  Area  Team 
and  Weapon  System  Team144 
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Major  Command  Mission  Area  Teams 

ACC 

Space  Control 

Accessions 

Combat  Delivery 

Force  Enhancement 

Education 

Precision  Employment 

Space  Forces  Support 

Contingency  Base  Operations 

Force  Application 

AFSOC 

Aerospace  Control 

Force  Application 

(Counterair/Theater  Missile  Defense) 

AMC 

Provide  Mobility  of  Forces  in 

Surveillance  & 

Airlift 

Denied  Territory 

Reconnaissance 

Air  Refueling 

Aviation  Foreign  Internal 

Theater  Battle  Management 

Defense 

Strategic  Air  Defense 

Joint  Air/SOF  Battlefield 

Counter  Information 

Interface 

Electronic  Combat 

Rescue 

PSYOPS 

AFSPC 

AETC 

Training 

Functional  Area  Teams 

Intelligence 

Modeling  and  Simulation 

Medical 

Logistics 

Weather 

Flight  Standards 

Personnel 

C4I 

Test  &  Evaluation 

Inspector  General 

Civil  Engineering 

Information  Warfare 

Table  21  Mission  by  Major  Command  and  Functional  Area  Teams 

Sample  MAT  membership 

Aerospace  Control  MAT  Membership 

DRAM  (MAP  Leader) 

DRSR  (Training  Rqmts) 

LGF16  (F-16  Logistics) 

DRAC  (BMC4I  Rqmts) 

SMO-22  (F-22  Rqmts) 

LGMA  (Acq  Logistics) 

DRAO  (F-15  Rqmts) 

SMO-P  (Mission  Ping  Rqmts) 

LGWM  (Munitions  Logistics) 

DRAW  (ABL/Weapons 

SMO-IC  (Interoperability 

SCCS  (Comm/Computer  Sys) 

Rqmts) 

Rqmts) 

SGPX  (Human  Factors) 

DRBT  (Special  Prgms) 

SMO-J  (JSTARS  Rqmts) 

XPFA  (Force  Structure) 

DRCT  (AWACS  Rqmts) 

DR-XP  MAST  (MAP  process) 

XPJD  (Doctrine) 

DRCP  (GTACS  Rqmts) 

DOTW  (Air-to-Air  Ops) 

XPJT  (Arms  Ctrl) 

DRFP(EW  Rqmts) 

DOXE  (Exercise  &  Ops 

PACAF/DOQ 

DRFR  (Airborne  Recce 

Mgmt) 

USAFE/DOQ 

Rqmts) 

DOYG  (BM  Ops  &  Tmg) 

AF/XORT 

DRFS  (Space  &  Intel  Rqmts) 

INAA  (Threat/Intel) 

SAF/AQPT 

DRMR  (RAMS) 

INXP  (Intel  Rqmts) 

*ASC/XRC  (TPIPT  Chief) 

DRPF  (F-16  Rqmts) 

LGF15  (F-15  Logistics) 

Table  22  Membership  on  the  Aerospace  Control  Mission  Area  Team 
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Air  Force  Material  Command  Technical  Planning  Integrated  Product  Teams 

ASC 

ESC 

Air-to-Surface 

Weather 

Electronic  Combat 

Theater  Battle  Management 

Aerospace  Control 

Recce/Surveillance/  Intell 

Special  Operations 

Strategic  Air  Defense 

Air  Base  Systems 

Modeling/Simulation 

Aircrew  Training 

Information  Warfare 

Mobility 

Rescue 

HSC 

SMC 

Environmental,  Safety,  & 

Space  Control 

Occupational  Health 

Force  Enhancement 

Human  Systems  Integration 

Space  Forces  Support 

Operational  Medical  Support 

Strategic  Deterrence 

Table  23  Membership  on  the  Aerospace  Control  Mission  Area  Team 

Sample  TPIPT  Membership 

Aerospace  Control  TPIPT  Membership 

ASC/XRC  (TPIPT  Chief) 
ASC/LAAT  (Attack  Ops) 
ASC/LPA  (Propulsion) 
ASC/LY  (Common  Avionics) 
ASC/RA  (Recce) 

ASC/RAX  (ISR) 
ASC/RWWZ  (EW  Support) 
ASC/SD  (Aircraft  PSO) 
ASC/VF  (F-15) 

ASC/YA  (A/A  JSPO) 
ASC/YFEX  (F-22) 
ASC/YPR1  (F-16) 

ESC/XRS 

ESC/ZJI  (Combat  ID) 

• 


HSC/XRS  (Human  Factors) 
SMC/XRT  (Missile  Warning) 
AL/CFPP  (Armstrong  Lab) 
PL/XP  (Phillips  Lab) 

RL/XP  (Rome  Lab) 

WL/XP  (Wright  Lab) 

Lincoln  National  Lab 
Los  Alamos  National  Lab 
MIT  Lincoln  Lab 
Sandia  National  Lab 
AFIWC/CA  (Info  Warfare) 
AFMC/DRJ  (Special  Programs) 
NSIA 
DPAS 

♦ACC/DRAM  (MAT  Leader) 


Table  24  Membership  on  the  Aerospace  Control  Technical  Planning  Integrated  Product 

Team 
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Figure  6  Modernization  Planning  Process  Flow147 


These  Mission  Area  Teams  follow  a  process  to  determine  what  the  current  capabilities  • 
are,  what  they  need  to  be,  and  how  best  to  fill  in  any  deficiencies  now  and  in  the  future.  The 
MATs  look  over  a  long  period  of  time,  projecting  our  capabilities  and  potential  enemy  forces  and 
capabilities  25  years  into  the  future.  The  process  that  is  used  starts  with  the  National  Military 
Strategy  and  uses  Mission  Area  Assessments  (MAA)  and  Mission  Needs  Analysis  (MNA)  to 
determine  current  and  future  mission  needs.  Mission  Solution  Analysis  (MSA)  is  then  used  to 
determine  preferred  solutions  and  operational  requirements  to  fill  deficiencies  in  current  or 
projected  capabilities. 


Mission  Area  Assessment  -  Mission  Area  Assessments  are  an  analysis  of  the  ability  to  undertake 
the  assigned  tasks  by  the  Unified  CINCs  and  the  major  commands  defined  by  the  concept  of 
operations  identified  for  a  military  contingency  plan.  Through  a  process  known  as  strategy-to- 


147  From  AQ/XO  briefing  on  Planning. 
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Mission  Area  Plans  -  Mission  Area  Plans  are  then  developed  from  the  Mission  Solution 
Analysis.  Mission  Area  Plans  cover  a  period  of  25  years  and  use  the  results  of  the  Mission  Area 
Assessment  and  the  Mission  Needs  Analysis  to  “document  the  most  cost  effective  means  of 
correcting  task  deficiencies  from  among  non-material  solutions,  changes  in  force  structure, 
system  modification  or  upgrades,  science  and  technology  applications,  and  new  acquisitions”  15°. 
The  Mission  Area  Plans  are  comprised  of  individual  weapon  system/capability  modernization 
roadmaps  outlining  the  modernization  plan  for  each  weapon  system  in  the  mission  area.  An 
example  of  a  weapon  system  modernization  roadmap  is  shown  in  the  figure  below.  These 
Mission  Area  Plans  are  developed  by  the  Mission  Area  Teams.  The  structure  for  this  Mission 
Area  Plan  is  detailed  in  AFI 10-1401.  Functional  Area  Plans  cover  the  cross  cutting  systems  that 
support  multiple  MAJCOMs  and  services  and  follow  the  same  basic  process.  The  next  step 
following  the  development  of  the  Mission  Area  Plans  is  to  limit  the  solution  set  in  the  mission 
area  plans  with  the  real  fiscal  constraints. 

The  culmination  of  the  modernization  planning  process  is  the  Four  Star  Review  of  senior 
Air  Force  leaders.  Here  the  various  mission  area  and  functional  area  plans  are  reviewed  and 
approved  and  fiscal  constraints  are  taken  into  account.  Here  the  potential  solutions  included  in 
the  proposed  MAPs  are  “racked  and  stacked”  to  determine  which  will  be  pursued  and  funded 
and  which  will  not  due  to  fiscal  constraints.  The  panel  is  briefed  by  the  mission  area  team 
members.  Following  the  review,  the  finalized  mission  area  plans  are  developed,  and  they 
identify  all  of  the  Major  Commands’  intentions  for  the  current  planning  cycle.  Projects  are 
projected  for  the  25  years  included  in  the  mission  area  plans. 

From  the  updated  MAPs  will  be  developed  a  list  of  systems  to  undergo  modification  and 
a  list  of  programs  to  be  included  in  the  major  commands’  program  Objective  Memorandum 
(POM).151  In  addition,  direction  for  science  and  technology  efforts  and  doctrine  and  tactics 


150  AFI  10-1401 

151  Program  Objective  Memorandum  (POM)— A  biennial  memorandum  submitted  to  the  Secretary  of  Defense 
(SECDEF)  from  each  Military  Department  and  Defense  agency.  It  proposes  total  program  requirements  for  the  next 
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changes  will  be  approved.  Material  based  solutions  will  begin  the  acquisition  process; 
operational  requirements  documents  will  be  drafted  and  program  offices  will  begin  the  pre¬ 
acquisition  planning.  Non-material  solutions  will  be  implemented  by  the  major  commands.  The 
MAJCOM  POMs  forecast  the  start  of  a  new  development  effort  by  including  the  development 
effort  with  its  expected  funding  levels  in  the  POM. 


F-15C  Modernization  Roadmap 
(Sample  Programs) 


Planned/Potential  Enhancements 


Name 

Description 

Status 

Downsized 

Tester 

Fast,  more  reliable  fault  analysis 

Funded 

MSiP 

Avionics  Upgrade  -  required  for  AMRAAM 

Funded 

Sec  Pwr 

Replace  CGB,  AWIAD,  etc. 

Unfunded 

APG-63M 

Incorporation  of  the  APG-63M  Radar 

Funded 

Data  Link 

Expanded  Situational  Awareness 

Unfunded 

HMS 

Helmet  Mounted  Sight  for  cuing 

Unfunded 

Tails/Spars 

New  Design  -  Upgrade  to  8,000  Flight  Hours 

Funded 

6  years.  It  includes  rationale  for  planned  changes  from  the  approved  Future  Years  Defense  Plan  (FYDP)  baseline 
within  the  fiscal  guidance  issued  by  the  SECDEF.  (AFM  11-1) 
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Figure  7  Sample  Modernization  Roadmap  for  the  F-15C  Weapon  System152 


The  list  of  unfunded  programs  contained  in  the  mission  area  plans  are  used  to  document 
the  unfunded  needs  of  the  major  commands  and  the  Air  Force  which  are  used  as  input  into,  the 
development  of  future  Air  Force  positions  and  justification  for  input  into  defense  planning 
guidance  which  allocates  the  expected  defense  resources  to  the  respective  services. 


Role  of  Defense  Contractors  in  the  Planning  Process 

During  the  modernization  planning  period  the  industry  attempts  to  influence  the  outcome 
of  the  planning  process  through  marketing  activities  at  all  levels,  from  the  four  star  level  to  the 
junior  captains  at  the  program  office  and  requirements  officer  level.  Contractor  input  is  directly 
sought  through  concept  calls  where  companies  and  individuals  propose  different  solutions  to 
problems.  An  industry  wide  representative  of  the  industrial  associations  participates  on  the 
TPIPTs. 

Individual  companies  naturally  try  to  influence  the  selection  of  the  “preferred  solutions” 
through  marketing  activities  at  all  levels.  Their  objective  is  to  assure  that  their  company  can 
compete  and  be  selected  for  as  many  of  the  funded  projects  as  possible.  And  they  want  to  ensure 
that  the  program  definition  at  the  earliest  stages  dictate  requirements  that  only  their  specific 
companies  have  the  solution  for  and  hence  the  greatest  chance  of  being  selected  in  any 
subsequent  acquisition  effort.  Industry  also  listens  to  gain  early  insight  into  which  expected 
development  programs  or  modifications  may  be  selected  so  as  to  get  an  early  start  on  preparing 
proposals  and  technology  development  efforts  to  gain  an  upper  hand  in  any  subsequent  selection 
process.  Company  representatives  are  not  directly  part  of  the  Mission  Area  Teams  or  the 
Technology  Planning  Integrated  Product  Teams  but  seek  to  influence  the  outcomes  of  the 
Mission  Solution  Analysis  by  providing  ideas,  information,  and  analysis  on  various  ideas  to 
individuals  at  the  program  offices,  requirements  officers,  and  at  the  senior  levels. 


152 


AFI 10-1401 
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B.  Directed  Programs 

An  alternative  method  to  the  Air  Force  modernization  planning  process  for  starting  a  new 
program  is  a  directed  program.  Here  a  senior  leader  directs  the  beginning  of  a  system  that  may 
not  have  been  included  in  prior  mission  plans  or  in  a  MAJCOM  POM.  This  top  down  vice 
bottom  up  process  appears  to  be  used  frequently.  These  programs  may  be  the  result  of  a  new 
technology,  a  high  visibility  identified  deficiency,  or  personal  interest  of  a  national  or  service 
leader.  These  directed  programs  will  be  incorporated  into  future  Mission  Areas  Plans  and  the 
POM  planning  cycles  for  the  major  commands.  In  this  situation,  a  mission  needs  statement  or 
operational  requirement  document  is  generated  referencing  the  senior  leader’s  direction. 

New  concept  weapons  and  systems  typically  start  through  this  process  as  opposed  to  the 
official  planning  process  where  the  staff  processes  by  their  nature  of  compromise  and  consensus 
lead  to  incremental  solutions  and  do  not  lend  themselves  to  far  reaching  innovative  steps.  Some 
of  the  systems  that  have  been  started  as  directed  programs  include  the  Airborne  Laser  Program, 
the  Joint  Direct  Attack  Munition  (JDAM),  and  the  Joint  Strike  Fighter.  Other  non-revolutionary 
programs  also  start  through  this  process  such  as  the  Tactical  Reconnaissance  Pods  (TAC  RECCE 
POD),  began  when  operational  deficiencies  were  identified.  Little  or  no  data,  other  than 
anticdotal,  exists  as  to  the  percentage  of  programs  that  started  as  result  of  the  formal  planning  vs 
the  directed  programs. 
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Requirement  Process 


A.  Mission  Needs  Statement  and  Operational  Requirements  Document 
Generation  Process 

One  outcome  of  the  mission  planning  process  are  identified  mission  needs  and  a  set  of 
preferred  solutions.  The  mission  need  statement  and  the  operational  requirements  document 
generation  process  is  designed  to  turn  operational  deficiencies  from  the  Mission  Needs  Analysis 
into  a  set  of  operational  requirements  that  are  the  basis  for  the  specifications  and  requirements  of 
a  development  effort.  These  are  the  first  steps  in  the  formal  acquisition  process.  Decisions  must 
be  made  as  to  how  to  fulfill  the  need,  how  the  system  should  work,  and  what  the  operational 
characteristics  of  the  system  should  be.  This  is  done  through  a  process  of  evaluating  alternatives, 
performing  cost  and  operational  effectiveness  analyses  (COEAs),  determining  what  the 
operational  parameters  for  the  specific  system  are,  and  then  finally  the  development  of  the 
operational  requirement  document  and  shepherding  it  through  the  approval  process.  This  process 
begins  with  the  various  outputs  from  the  planning  process,  specifically  the  Mission  Area 
Assessment,  the  Mission  Need  Analysis,  and  the  Mission  Solution  Analysis. 
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REQUIREMENTS  GENERATION 


MAA 


MISSION  ESSENTIAL  TASK  LIST 
&  TECH  OPPORTUNITES 


(MNA) 


Milestone  0 


Training 


EFICIENCES 


MATERIAL 

SOLUTION 


T  3  ctics 

NON-MATERlAL  SOLUTIONS 


(MSA) 


N!*D  CONOPS 
y  •CONCEP 

•CONCEP 

•CONCEP 

CRD 


Milestone  1 


ACQUISITION 


ACQUISITION 


Figure  8:  Mission  Needs  Statement  and  Operational  Requirements  Document  Relationship 
with  the  Planning  and  Acquisition  Processes 


Mission  Needs  Statements  -  Mission  Needs  Statements  document  the  capability  required  to 
accomplish  a  certain  operational  task  and  an  inability  to  fulfill  the  requirement  with  a  change  in 
training,  tactics,  or  other  non-materiel  solutions. 

The  mission  needs  statements  includes  six  sections  containing  information  that  defines 
the  need  and  potential  ways  tomeet  it.  The  first  section  identifies  the  mission  area  and  long  range 
planning  objectives  that  it  addresses  and  how  the  requirement  fits  with  the  long  range  investment 
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planning.  Section  two  identifies  the  current  or  predicted  mission  deficiency  and  the  threat 
projections  upon  which  it  is  based.  Section  three  discusses  potential  non-material  solutions  and 
describes  why  these  are  inadequate.  Section  four  discusses  potential  material  solutions  and 
identifies  a  range  of  alternatives.  It  also  indicates  initial  preferences  for  emphasis.  These 
alternatives  become  the  basis  for  the  concept  studies  that  are  performed  after  Milestone  0. 

Section  five  details  the  constraints  and  interfaces  placed  on  any  system  to  meet  the  need  and  the 
operational  environments  that  the  system  is  expected  to  operate  in.  The  last  section  indicates  the 
potential  for  joint  service  use. 

Once  a  Mission  Needs  Statement  is  submitted  it  must  pass  through  an  approval  and 
validation  process.  Depending  on  the  scope  and  the  potential  cost  of  the  solutions,  mission  needs 
statements  are  approved  at  the  Joint  Operational  Requirements  Council  at  the  Vice  Chief  of  Staff 
level  or  the  Air  Force  Operational  Requirements  Council,  its  Air  Force  level  equivalent.  A 
validated  Mission  Need  Statement  is  required  for  a  Milestone  0  Decision  that  allows  for  early 
studies  of  alternatives  and  operational  and  cost  implications  of  a  new  acquisition  system.  The 
Mission  Needs  Statements  are  developed  by  the  MAJCOMs  and  document  the  deficiency  and 
propose  a  preferred  way  of  meeting  that  deficiency.  These  are  generated  from  the  outcome  of  the 
Mission  Needs  Analysis  and  the  modernization  planning  process. 


Milestone  0  -  Concept  Studies  Approval 

A  validated  Mission  Needs  Statement  is  required  for  a  Milestone  0  decision.  The 
Milestone  0  decision  is  made  by  the  Milestone  Decision  Authority  (MDA),  either  the 
OUSD(A&T)  or  SAF/AQ  depending  on  the  size  of  the  program.  Following  a  Milestone  0 
decision  an  acquisition  decision  memorandum  (ADM)  will  be  generated  and  a  Program 
Management  Directive  (PMD)  will  be  issued  directing  the  major  commands  to  complete  the 
Phase  0  activities. 


Phase  0  -  Concept  Exploration  and  Definition  Activities 
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Phase  0  is  the  time  when  the  specifics  of  the  system  to  be  developed  is  determined.  From 
a  validate  Mission  Needs  Statement  potential  solutions  are  identified.  Studies  are  run  and  a  cost 
and  operational  effectiveness  analyses  is  performed.  From  this  an  operational  requirements 
document  is  written,  and  then  forms  the  basis  for  a  subsequent  acquisition  effort.  Also  during 
Phase  0  the  acquisition  process  is  started.  Acquisition  related  steps  will  be  discussed  in  a  later 
section. 


Analysis  of  Alternatives 

Following  a  Milestone  0  decision  much  of  the  analysis  of  alternatives  is  accomplished  in 
the  Mission  Solution  Analysis  and  its  purpose  its  to  identify  alternative  methods  of  meeting  the 
operational  requirements  and  meeting  mission  deficiencies.  Here  a  Concept  Action  Group 
(CAG)  is  established  to  study  the  alternative  concepts.  The  results  of  these  concept  studies  are 
used  to  prepare  the  Cost  and  Operational  Effectiveness  Analysis. 


Cost  and  Operational  Effectiveness  Analysis  (COEA) 

The  purpose  of  the  Cost  and  Operational  Effectiveness  Analysis  is  to  assist  decision 
makers  in  selecting  the  most  cost  effective  material  method  to  satisfy  a  mission  need.  It  is 
designed  to  compare  several  solutions  on  the  basis  of  cost  and  operational  effectiveness  and 
document  the  rationale  for  choosing  the  preferred  solution.153  COEAs  are  required  for  all 
potential  AC  AT  I  programs  and  other  programs  designated  by  the  Milestone  Decision  Authority. 
They  are  not  typically  required  for  ACAT II,  III,  and  IV  programs. 

There  are  two  phases  of  the  COEA:  COEA  I  to  support  Milestone  I  and  COEA  II  to 
support  Milestone  II.  COEA  I  for  Milestone  I  is  based  on  the  Mission  Needs  Statement  and 
builds  on  the  mission  area  assessment  and  the  mission  needs  analysis.  COEA  I  evaluates  a  broad 
range  of  alternative  concepts  to  meet  the  operational  need.  An  example  would  be  the  need  for  a 
reconnaissance  capability  and  potential  alternative  solutions  may  be  manned  aircraft,  unmanned 
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aircraft,  and  satellites.  This  COEA  identifies  the  performance,  operational  characteristics,  and 
capabilities  most  affecting  the  outcome  of  the  campaign  so  program  design  and  cost  objectives 
can  be  established  for  the  Milestone  I  decision.  COEA  II  is  performed  during  the  Demonstration 
and  Validation  phase  of  a  program  after  program  initiation  at  Milestone  I.  In  COEA  II  specific 
hardware  alternatives  are  analyzed  such  as  should  an  aircraft  have  one  or  two  engines  or  have 
one  or  two  crew  members. 

The  COEAs  are  organized  and  executed  by  the  major  commands  with  assistance  from  the 
other  acquisition  and  testing  communities  as  required.  Each  COEA  is  defined  by  an  approved 
COEA  Study  Plan  which  identifies  the  parameters  for  the  study.  The  plan  identifies  the 
functional  objectives,  the  measures  of  performance,  and  the  measures  of  effectiveness  to  be 
evaluated.  It  also  defines  the  threat,  the  sources  of  data,  and  the  models  to  be  used  in  the  analysis 
as  well  as  the  schedule  and  resources  available  for  the  study.  For  major  programs,  a  COEA 
Oversight  Group  (COG)  and  a  Technical  Advisory  Group  (TAG)  are  established  to  oversee  the 
methods  and  assumptions  used  by  the  major  command  studies.  These  organizations  ensure  the 
COEA  plans  represent  the  senior  leadership’s  consensus  on  the  analytical  foundations  before 
submitting  it  to  the  approval  authorities  and  DoD  organizations154.  The  COEA  plan  is  reviewed 
in  draft  by  the  COG  and  the  TAG  prior  to  its  formalization  and  approval.  The  COG  is  chaired  by 
the  Vice  Chief  of  Staff  and  composed  of  representatives  from  the  interested  organizations.  The 
TAG  is  composed  of  technical  and  cost  analysts  and  supplies  the  COG  chairman  with  analysis  on 
the  content  and  completeness  of  the  COEA  plan  and  the  COEA  results. 

Operational  effectiveness  of  different  operational  concepts  is  accomplished  by  modeling 
and  simulation  organizations.  This  process  also  identifies  the  key  performance  parameters  that 
provide  the  system  its  effectiveness.  These  are  documented  in  Measures  of  Effectiveness  and 
Measures  of  Performance  which  will  be  used  in  the  development  of  the  Operational 
Requirements  Document,  the  Acquisition  Program  Baseline  and  the  Test  and  Evaluation  Master 
Plan. 
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Cost  and  performance  expectations  are  estimated  using  parametric  estimating  techniques 
into  broad  intervals  ranging  from  low  to  high  values.  Rough  cost  estimates  are  developed  for 
development,  operations,  support,  and  disposal  costs.  The  COE  A I  is  used  to  support  the 
selection  of  the  preferred  solution  and  is  used  to  establish  the  program  baseline  and  the  early 
acquisition  planning  activities  for  Milestone  I.  The  analysis  is  accomplished  when  the  concepts 
have  been  sufficiently  demonstrated  into  point  estimates  using  estimating  techniques  appropriate 
for  the  different  levels  of  maturity. 155 

The  COEA  is  often  performed  under  contract  by  one  of  many  defense  service  contractors 
that  regularly  support  and  are  under  contract  for  such  activities.  The  COEA  is  reviewed  in  draft 
form  by  interested  organizations  at  the  deputy  chief  of  staff  level  functions  in  the  Pentagon.  It  is 
then  reviewed  and  validated  by  the  MAJCOM  commander  prior  to  its  formal  review  by  the 
COG.  The  COG  then  forwards  the  COEA  with  its  comments  to  the  Chief  of  Staff  who  endorses 
it  prior  to  its  approval  by  the  Milestone  Decision  Authority.  During  the  time  when  the  COEA  is 
being  accomplished  the  service  is  establishing  the  projected  cost  of  the  system  through  another 
group  of  financial  analysts  in  the  SAF/FM  organization.  The  SAF/FM  Cost  Analysis  Group 
(CAG)  develops  the  Service  Cost  Position  for  the  development  of  the  system.  If  differences  exist 
between  the  two  groups’  cost  estimates  the  differences  are  rectified.  No  discussion  of 
development  schedule  considerations  are  discribed  in  the  description  of  the  COEA  Process  in 
the  Air  Force  instructions. 

Following  the  COEA  Process,  the  next  step  is  the  development  of  the  Operational 
Requirements  Document. 


B.  Operational  Requirements  Document 

The  Operational  Requirements  Document  (ORD)  is  the  central  document  that  describes 
the  concept  of  operations  and  what  the  new  system  is  supposed  to  accomplish.  It  captures  the 
results  of  the  key  performance  parameters,  the  measures  of  effectiveness,  and  the  measures  of 
performance  from  the  COEA  and  other  analyses  to  form  the  basis  for  the  requirements  for  the 
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subsequent  acquisition  efforts.  The  Operational  Requirements  Document  establishes  both 
threshold  and  objectives.  No  acquisition  effort  can  proceed  through  Milestone  I  without  a 
validated  Operational  Requirements  Document. 

The  initial  Operational  Requirements  Document  used  for  the  Milestone  I  decision 
contains  general  information  that  is  required  for  Milestone  I.  Subsequent  versions  of  the  ORD 
at  Milestone  II  and  III  contain  additional  information  as  the  system  is  further  defined.  The 
progression  of  these  requirements  is  documented  in  the  Requirements  Correlation  Matrix  (RCM) 
that  tracks  the  changes  of  the  requirements  over  time  and  documents  the  reasons  for  the  changes. 

The  ORD  defines  the  general  system  characteristics  and  capabilities  needed  to  satisfy  the 
needs  documented  in  the  Mission  Needs  Statement.  If  defines  the  system  capabilities  such  as 
range,  lethality,  and  maneuverability  and  the  characteristics  such  as  weight,  size,  and  shape. 
Critical  characteristics  that  have  typically  driven  cost,  such  as  stealth  and  electromagnetic  pulse 
hardening,  are  also  defined.  Both  threshold  (the  minimum  acceptable  levels)  and  objectives  (the 
desired  levels)  are  defined.  Key  performance  parameters  are  defined  which  if  the  system  can  not 
meet,  then  the  continuation  of  the  program  must  be  reviewed.  These  key  performance 
parameters  are  included  in  the  Acquisition  program  baseline  and  are  reviewed  at  the  milestone 
decision  points. 

The  Operational  Requirements  Document  is  developed  by  the  Major  Commands  using 
the  results  of  the  COEA  studies  and  operational  analysis.  The  ORD  has  eight  sections.  Section 
one  describes  the  operational  capabilities  and  is  an  overview  of  the  system  requirement,  mission 
needs,  and  the  process  used  to  direct  the  development  of  the  ORD.  Section  two  identifies  the 
threat  that  the  system  is  designed  to  counter.  For  major  programs  this  references  a  Strategic 
Threat  Assessment  Report  generated  by  the  intelligence  community.  Section  three  identifies  the 
shortcomings  of  the  existing  systems  and  why  they  can  not  meet  the  need. 

Section  four  identifies  the  capabilities  required.  Listed  are  the  capabilities  and 
charateristics  of  the  system  desired.  Measures  of  effectiveness  and  performance  threshold  and 
objective  values  are  included.  Section  five  describes  the  logistics  support  system  for  the 
proposed  system  including  the  maintenance  planning,  support  equipment,  manpower  and  training 
and  force  structure  requirements,  and  other  logistics  requirements.  Section  6  identifies  the 
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infrastructure  and  interoperability  requirements  such  as  basing  and  adresses  how  C3I 
considerations  will  be  integrated.  Section  7  identifies  the  number  of  systems  needed  for  both 
operations  and  training.  The  last  section  identifies  schedule  considerations.  It  defines  what 
actions  when  completed  signify  initial  operational  capability  and  full  operational  capability 
indicating  the  system  is  ready  for  use.  It  will  also  state  “if  availability  in  a  specific  time  frame  is 
important”  and  will  specify  an  objective  IOC  and  “  state  the  impact  if  that  objective  is  not 
achieved.” 156 

Both  the  Operational  Requirements  Document  and  the  Mission  Needs  Statement  must  be 
validated  and  approved  to  be  used  for  a  milestone  decision. 


C.  MNS  and  ORD  Approval  Process 

For  a  potential  acquisition  program  to  proceed  past  Milestone  I  and  become  an 
acquisition  program  it  must  have  validated  and  approved  Operational  Requirements  Document. 
The  process  to  obtain  approval  is  designed  to  ensure  that  all  concerns  and  issues  are  dealt  with 
prior  to  the  release  of  the  document.  The  process  is  illustrated  in  the  figure  below.  The  major 
command  level  review  allows  the  functionals  within  the  major  command  to  comment  and  work 
issues  at  their  level  prior  to  its  review  at  the  Pentagon.  The  Pentagon  level  review  process  is  lead 
by  HQ  USAF/XOR,  the  Directorate  for  Requirements.  A  draft  document  is  reviewed  by  the 
requirements  review  council  which  has  representation  for  each  of  the  functional  areas. 

Following  the  resolution  of  issues  the  major  command  will  forward  a  final  ORD  with  the 
commanders’  signature.  HQ  USAF/XOR  and  the  Requirements  Review  Council  will  again 
review  the  final  document,  obtain  final  Deputy  Chief  of  Staff  level  review  from  the  functionals, 
and  submit  the  ORD  to  the  Air  Force  Operational  Review  Council,  a  two  star  level  review  by  the 
Deputy  Chiefs  of  Staffs.  The  finalized  document  is  then  sent  to  the  Chief  of  Staff  for  his 
approval.  Following  the  Chief  of  Staffs  approval,  depending  on  the  size  of  the  potential  project, 
the  ORD  is  sent  to  either  the  Joint  Operational  Requirement  Council  for  validation  and  then  to 
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the  Undersecretary  of  Defense  for  Acquisition  and  Technology  or  to  the  Assistant  Secretary  of 
the  Air  Force  for  acquisition  as  the  milestone  decision  authority. 
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Figure  9:  MNS  and  ORD  Approval  and  Validation  Process1*7 


157  Taken  from  AQ/XO  offsite  briefing  on  requirements  process. 
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Air  Force  Planning,  Programming  and 
Budgetting  System  (PPBS) 


A.  Resource  Allocation  Process  and  The  Planning,  Programming,  and 
Budgeting  System 

The  resource  allocation  and  the  Planning,  Programming,  Budgeting  System  are  annual  or 
biannual  processes  used  to  authorize,  allocate  resources,  and  develop  Air  Force  and  DoD  budgets 
for  all  activities.  The  programming  process  documents  and  authorizes  every  activity  in  the  Air 
Force.  The  resource  allocation  process  is  part  of  this  process  and  determines  the  apportionment 
of  dollars,  manpower,  and  equipment  resources  for  each  activity.  The  budgeting  process  takes 
the  results  of  the  resource  allocation  process  and  develops  the  annual  budget  requested  from 
Congress.  Once  the  budget  is  passed  it  then  distributes  and  tracks  the  money  received.  The 
PPBS  system  dictates  many  of  the  actions.  The  Air  Force  uses  its  corporate  process  to  make 
programming  and  resource  allocation  decisions.  Through  these  processes  the  money  and 
resources  available  to  a  development  effort  and  all  other  service  activities  are  decided. 


A.l.  Planning  Programming  and  Budgeting  System 

The  basis  for  the  these  activities  is  the  Planning,  Programming,  and  Budgeting  System 
(PPBS)  which  is  directed  by  DoD  Directive  7045.14.  The  PPBS  system  was  started  in  1960 
under  then  Secretary  McNamara  to  try  to  bring  a  more  results  focused  process  to  defense 
budgeting.  The  PPBS  system  is  designed  to  provide  an  orderly  process  with  which  to  allocate 
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the  resources  available  to  the  DoD  in  order  to  achieve  the  greatest  military  capacity.  At  the 
foundation  of  this  process  is  the  Future  Years  Defense  Plan  (FYDP).  Future  Years  Defense 
Program  is  the  official  document  that  summarizes  Secretary  of  Defense-approved  programs  for 
the  Department  of  Defense  (DoD).  The  FYDP  projects  detailed  resource  requirements  for  6 
years  and  force  structure  for  9  years.  During  the  PPBS  process,  the  FYDP  is  updated  three  times 
every  year  to  reflect  the  Services'  Program  Objective  Memo~andum,  Services'  Budget  Estimate 
Submission,  and  the  DoD  portion  of  the  President's  budget.158  The  FYDP  is  the  official  record  of 
all  financial  management  decisions  that  have  been  made  and  is  the  nucleus  of  the  DoD  resource 
management  system.159 

The  FYDP  is  structured  around  major  force  programs:  strategic  forces;  general  purpose 
forces;  intelligence  and  communications;  airlift/ sealift;  guard  and  reserve  forces;  research  and 
development;  central  supply  and  maintenance;  training;  administration;  and  support  of  other 
nations.  These  cross-service  major  force  areas  include  programs  from  all  services  and  DoD 
agencies  and  allow  policy  makers  to  see  the  resources  allocated  to  the  different  major  mission 
areas.  Ostensibly  the  FYDP  has  as  inputs  the  appropriations  from  Congress  that  should  result  in 
outputs  of  military  forces  and  capabilities;  however,  the  system  is  primarily  used  in  reverse  to 
plan,  develop,  and  justify  the  proposed  budget  sent  to  DoD  and  Congress  in  the  form  of  the 
President’s  Budget.  Following  passage  of  the  budget  the  funds  are  allocated  down  to  the 
program  element. 

Program  Element  is  the  lowest  subdivision  or  building  block  of  the  PPBS  system.  The 
program  element  represents  a  specific  activity  or  mission  and  contains  the  authorization  for  the 
forces,  the  equipment,  and  the  money  to  carry  out  that  mission.  There  are  nearly  a  thousand 
Program  Elements  in  the  Air  Force  alone,  each  representing  a  separate  activity.  Each  major 
development  activity  will  have  a  separate  program  element  specifying  its  resources,  funding,  and 
objectives.  The  management  of  the  program  elements  are  overseen  at  the  lowest  level  by  the 
program  element  monitors.  These  Captain,  Major,  and  Lieutenant  Colonel  level  action  officers 
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at  the  Pentagon  are  the  representative  of  their  program  elements  and  the  programs  they  represent 
at  the  Air  Staff  level.  They  are  the  primary  interface  between  the  PPBS  system  and  the  people 
carrying  out  the  mission  efforts.  For  acquisition  related  activities  most  PEMs  are  in  the  SAF/AQ 
organization  and  for  mature  weapon  systems  they  reside  in  the  HQ  USAF/XO  organization. 


Figure  10:  The  Structure  of  the  Future  Year  Defense  Plan  is  Separated  by  Major  Mission 
Area,  Military  Service,  and  Appropriations 

The  Air  Force’s  section  of  the  Future  Year  Defense  Plan  is  maintained  in  a  computerized 
database  system  known  as  ABIDES  (Automated  Budget  Interactive  Data  Environment  System) 
which  is  maintained  by  SAF/FMB.  The  ABIDES  Database  is  a  system  that  was  developed  in  the 
1970’s  and  is  difficult  to  use  and  requires  dedicated  people  with  specific  training  to  manipulate 
and  retrieve  data.  One  report  stated  that  ABIDES  “requires  thorough  user  training” .  It,  however, 
is  reportedly  in  the  process  of  being  updated.  This  system  is  used  to  run  various  exercises  to 
determine  the  impact  of  changes  in  funding  levels  on  the  Air  Force  in  the  development  of  the 
proposed  budget. 
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Figure  11:  The  Planning  Programming  and  Budgeting  Process  takes  two  years  to 
complete.  Above  is  the  rough  schedule  for  the  various  steps  in  the  process.160 


There  are  three  key  players  in  the  PPBS  system.  Planning  is  headed  by  HQ  USAF/XO 
and  is  expected  to  assess  the  threat  and  determine  resources  and  forces  needed  to  meet  national 
objectives.  Programming  is  headed  by  HQ  USAF/PE  which  integrates  fiscal,  manpower,  and 
material  requirements  with  expected  available  resources.  Budgeting  is  led  by  SAF/FM  who 
refines  costs  estimates  for  the  proposed  program,  and  develops  the  proposed  Budget  Estimate 
Submission  (BES)  for  the  Air  Force. 


Many  of  the  longer  range  planning  aspects  have  already  been  discussed  in  the 
modernization  planning  process  which  result  in  the  MAJCOM  POM  inputs.  The  PPBS  planning 
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aspects  are  directly  tied  to  the  allocation  of  resources  where  the  modernization  planning  process 
is  designed  to  identify  the  best  technology  and  weapon  systems  for  the  future.  Planning  with 
respect  to  the  PPBS  system  produces  fiscal  forecast,  planning  guidance,  and  program  guidance.161 
During  the  planning  phase,  the  MAJCOMs  develop  program  objective  memorandum  describing 
their  plans  for  the  upcoming  budget  cycle.  This  process  takes  preliminary  high  level  guidance 
from  DoD  and  the  Air  Staff  and  develops  it  into  a  detailed  plan  for  forces  and  resources  to  carry 
out  the  specific  missions  each  Major  Command  is  assigned.  The  primary  input  into  the  Pentagon 
planning  is  the  MAJCOM  POM  input  detailing  the  expected  level  of  resources  required, 
preliminary  Defense  Planning  Guidance  from  OSD,  the  previous  year’s  approved  FYDP,  and  the 
MAJCOM  develop  Mission  Area  Plans  described  earlier.  In  the  MAJCOM  POM  inputs,  the 
MAJCOMs  identify  any  initiatives  which  are  new  proposals  for  additional  funds;  any 
disconnects  which  identifies  missions  that  are  unexecutable  because  a  mismatch  of  resources  and 
the  approved  program;  and  any  offsets  which  identify  resources  to  “  pay”  for  the  disconnects  and 
initiatives. 

The  MAJCOM  POM  inputs  provide  information  to  the  Chief  and  the  Secretary  with 
specific  and  detailed  information  to  support  the  Air  Force  position  in  the  development  of  the 
Defense  Planning  Guidance  at  the  OSD  level.  The  Defense  Planning  Guidance  specifies  the 
apportionment  of  defense  resources  and  funds  to  the  services  and  is  the  culmination  of  the 
planning  phase. 


B.  Programming 

The  programming  phase  begins  with  the  issuance  of  the  program  guidance  and  results  in 
the  service  Program  Objective  Memorandum  (POM)  and  the  OSD  Program  Decision 
Memorandum  (PDM).  This  process  develops  the  Air  Force  portion  of  the  DoD  Future  Year 
Defense  Program. 

The  Air  Staff  adjusts  the  content  of  Air  Force  programs  to  meet  content  and  resource 
levels  allocated  by  the  Defense  Planning  Guidance.  The  Air  Staff  then  updates  the  AF  planning 
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and  program  guidance  that  details  the  proposed  force  structure  and  resource  allocations  within 
the  DPG  levels  to  the  Major  Commands.  The  MAJCOMs  identify  any  new  initiatives, 
disconnects,  and  offsets  that  resulted  from  the  expected  levels  and  that  result  from  the  defense 
planning  and  Air  Staff  programming  process. 

The  decision  process  in  the  programming  phase  is  the  resource  allocation  process.  The 
resource  allocation  process  links  the  planning,  the  programming,  and  the  budgeting  process.  The 
outcome  is  the  Program  Objective  Memorandum  that  details  the  Air  Force  proposed  program  and 
is  forwarded  to  OSD.  The  completion  of  the  POM  process  with  it  associated  resource  allocation 
process  is  the  primary  decision  point  for  most  programs’  authorized  activities  and  budgets.  The 
new  process  that  is  used  to  make  these  decisions  at  the  Air  Force  level  is  the  Air  Force  Corporate 
Process.  The  final  POM  decisions  are  made  by  the  Secretary  and  the  Chief  of  Staff  receiving  the 
recommendation  from  the  Corporate  Process. 

B.l.  The  Enhanced  Air  Force  Corporate  Process 

The  current  Air  Force  Corporate  Process  is  an  enhancement  of  the  previous  corporate 
structure  and  involves  more  people  at  each  level.  The  previous  process  involved  an  Air  Force 
Council  and  Air  Force  Board  which  reviewed  the  results  of  numerous  resource  allocation  teams. 
The  enhanced  structure  has  further  developed  this  process  to  include  five  levels  and  formalizes 
many  of  the  aspects  that  were  previously  accomplished  on  an  ad  hoc  basi.  It  is  intended  to 
provide  the  senior  leadership  with  the  “  corporate  position  while  retaining  the  responsibilities  of 
the  functional  organizations” . 162 

The  enhanced  Corporate  Process  of  5  levels  of  organizations  with  final  review  by  the 
Chief  of  Staff  and  the  Secretary.  These  are  depicted  in  the  figure  below. 


162  AF  Corporte  Process  draft  writeup.  Obtained  from  AF/XPP 
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The  Air  Force  Corporate  Structure 


Functional  Organizations 
XO  AQ  LG  SC  DP  SN  FM  CE. 


Corporate  Review  Structure 
SECAF/CSAF 


Figure  12:  The  Enhanced  Air  Force  Corporate  Structure 

The  Air  Force  Council  -  The  Air  Force  Council  is  a  four  star  level  review.  It  consists  of  the 
senior  leaders  from  the  various  functional  areas  and  major  commands.  This  group  approves  the 
Air  Force  corporate  position,  which  is  then  presented  to  the  Chief  of  Staff  and  the  Secretary  for 
review. 

The  Air  Force  Board  -  The  Air  Force  Board  is  a  two  star  level  review  of  senior  leaders.  The 
enhanced  corporate  structure  enlarged  the  Air  Force  board  to  include  additional  functional  areas. 
It  now  consists  of  23  core  functional  areas  and  7  other  members. 

The  Air  Force  Group  -  The  Air  Force  Group  is  the  first  corporate  integrated  review.  It  consists 
of  23  core  members  and  7  other  members  representing  the  functional  areas  of  the  Air  Force. 
Members  are  colonels  or  civilian  equivalents.  During  the  POM  process  this  group  is  empowered 
with  “  off-the-table”  decision  authority  as  it  brings  forward  ideas  to  the  Air  Force  Board.  This 
group  reviews  the  program  options  for  compliance  with  Air  Force  guidance;  validates  cost, 
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schedule  and  completeness  of  the  program  options;  develops  new  options  as  needed;  entertains 
new  initiatives  to  meet  core  competencies  of  the  Air  Force  and  receives  briefings  as  necessary  to 
make  informed  decisions  on  programs.  This  group  does  much  of  the  preliminary  screening  of 
programs  before  the  Air  Force  Board. 

Group  members  are  assigned  to  and  reside  in  their  functional  organizations  and  are 
linked  electronically.  No  additional  manpower  authorizations  are  provided. 


Mission  and  Mission  Support  Panels  -  Ten  new  panels  were  developed  by  realigning  the  Air 
Force  Resource  Allocation  Teams  with  the  core  missions  and  supporting  functions  in  the  Air 
Force.  These  panels  serve  as  the  Air  Force  “  Centers  of  Expertise”  and  represent  the  first  level  of 
corporate  review  of  the  mission  areas.  Panel  membership  consists  of  cross  staff  functional 
expertise  from  HQ  USAF  organizations  to  ensure  that  all  aspects  of  a  proposal  are  thoroughly 
evaluated  before  being  presented  to  the  Air  Force  Group.  While  retaining  a  corporate 
perspective  for  programs  within  their  mission  areas,  Panels  must  still  play  the  role  of  advocate 
within  the  corporate  process. 

Panel  members  are  assigned  to  and  reside  in  their  functional  organizations  and  are  linked 
electronically.  No  additional  manpower  authorizations  are  allowed. 


Mission  Panels 

Air  Superiority 
Power  Projection 
Global  Mobility 
Information  Dominance 
Space  Superiority 


Mission  Support  Panels 

RDT&E 

Logistics 

C4I 

Installation  Support 
Personnel  and  Training 

Other  Panels 

National  Foreign  Intelligence 
Special  Access  Required 


Table  25:  Mission,  Mission  Support,  and  Other  Panels  in  the  Air  Force  Enhanced  Corporate 
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Integrated  Product  Teams  (IPT)  -  The  lowest  level  of  the  corporate  process  is  the  weapon  system 
level  IPT.  This  IPT  formalizes  the  informal  process  previously  used  within  the  Air  Staff.  The  IPT 
is  the  single  point  of  contact  to  the  major  commands  for  their  specific  programs.  IPT  membership 
consists  of  the  applicable  program  element  monitors,  and  representatives  from  each  functional 
organizations  experts  who  have  interests  or  responsibilities  in  the  program,  process,  or  issue.  The 
IPT  members  speak  on  behalf  of  their  functional  organizations  within  the  constraints  placed  on 
them  by  their  parent  organization.  The  IPT  chief  is  the  designated  Lead  PEM  or  “  Program 
Monitor” .  There  are  expected  to  be  about  70  separate  weapon  system  IPTs. 

The  IPTs  were  developed  in  response  to  DoD  guidance  to  use  IPTs  in  the  Acquisition 
programs  and  the  AFMC  suggestion  for  a  single  point  of  contact  for  weapon  systems.  IPT 
members  are  assigned  to  and  reside  in  their  functional  organizations  and  are  linked  electronically. 
No  additional  manpower  authorizations  will  be  provided. 
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From  Draft  Corporate  AF  Process  Document 
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This  corporate  review  process  results  in  the  Program  Objective  Memorandum  (POM)  which  is 
approved  by  the  Secretary  and  the  Chief  of  Staff  and  submitted  to  OSD.  This  represents  what 
actions  the  Air  Force  requests  to  do  for  the  years  covered  by  the  POM.  The  POM  is  submitted  to 
OSD  in  March  and  covers  a  time  period  2  to  1 1  years  in  the  future.  This  then  begins  the  OSD  level 
review  process  s 

OSD  Level  POM  Review  Process 

The  proposed  Air  Force  POM  is  reviewed  by  the  Commanders  in  Chief  of  the  Unified 
Commands,  the  Joint  Chiefs  of  Staff,  and  the  Defense  Planning  and  Resources  Board.  The  CINC’s 
comments  are  included  in  the  Chairman’s  Program  Assessment  which  he  provides  the  Secretary  of 
Defense  with  an  assessment  of  the  program  risk  associated  with  the  submitted  POMs,  the  JCS  views 
on  balance  between  the  services,  and  the  JCS  recommendation  on  how  to  achieve  improved  defense 
capabilities  within  the  funding  levels  directed  by  the  OSD  and  a  mobility  force  analysis.  OSD  then 
initiates  a  review  of  the  entire  DoD  program  and  with  the  assistance  of  the  Defense  Planning  and 
Resources  Board  identifies  alternatives  and  approaches  for  major  issues  identified  at  that  level. 

During  this  process  there  are  many  budget  exercises  at  the  Air  Staff  level  as  they  work  to 
respond  and  justify  the  Air  Force  program  to  various  requests  from  the  OSD  staff.  “  What  if’  drills 
with  large  program  resource  swings  and  very  short  suspenses  are  common  throughout  the 
development  of  the  Air  Force  POM  and  the  DoD  PMD.  The  primary  person  responding  to  these 
drills  is  the  Program  Element  Monitor,  the  associated  IPT,  and  the  Mission  Panels. 

The  Secretary  of  Defense  then  issues  a  Program  Decision  Memorandum  (PDM)  for  each 
service  that  documents  his  decision  as  the  authorized  programs  for  each  service.  The  Program 
Decision  Memorandum  limits  the  items  the  services  can  include  in  the  development  of  their 
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proposed  budgets.  This  is  done  to  limit  the  expected  programs  to  what  DoD  can  realistically 
afford.164 
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Figure  14:  OSD  POM  Review  Process165 


The  issuance  of  the  Program  Decision  Memorandum  (PDM)  marks  the  end  of  the 
programming  process  and  the  beginning  of  the  budgeting  Process.  Changes  are  made  to  the  Air 
Force  FYDP  as  directed  by  the  PDM. 


164  Przemieniecki,  J.S.  Editor.  Acquisition  of  Defense  Systems.  Washington  D.C.:  American  Institute  of  Aeronautics 
and  Astronautics.  1993. 

165  Przemieniecki,  J.S.  Editor.  Acquisition  of  Defense  Systems.  Washington  D.C.:  American  Institute  of  Aeronautics 
and  Astronautics.  1993.  Pg.  124 
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C.  Budgeting  Process 

Following  the  Program  Management  Decisions,  what  the  services  will  do  has  been  decided 
by  the  Secretary  of  Defense.  Now  begins  the  development  of  the  budget  which  requests  the  funds 
from  Congress  to  carry  out  the  program  that  has  been  agreed  to.  The  budget  process  includes 
several  more  reviews  and  major  decision  points  and  is  likely  to  program  change  programsas  it 
progresses  forward.  Reviews  at  the  Air  Force,  OSD,  OMB,  resident,  and  congressional  committees 
still  lie  ahead. 


Air  Force  Budget  Preparation 

Developing  the  Air  Force  budget  for  submission  to  OSD,  the  President,  and  the  congress  is 
the  role  of  the  SAF/FM  organization.  This  organization  takes  the  information  contained  in  the 
approved  PDM  and  the  updated  FYDP  in  the  ABIDES  database  and  refines  the  pricing  models  and 
updates  the  information  with  the  required  OSD  guidance,  economic  models,  current  inflation 
predictions,  and  commodity  prices.  They  then  develop  the  Budget  Estimate  Submission  (BES)  for 
the  approved  program  while  staying  within  DoD  fiscal  guidance.  The  program  content  is  adjusted 
to  keep  the  total  budget  request  within  the  DoD  guidance.  The  BES  details  the  cost  of  the  approved 
program  for  the  planned  year.  The  BES  is  submitted  to  OSD  in  September,  a  year  prior  to  its 
enactment. 


OSD  and  OMB  Review 

The  BES  is  then  combined  with  those  from  the  other  services  and  DoD  agencies  to  form  the 
Defense  Budget.  This  budget  is  then  coordinated  with  the  Office  of  Management  and  Budget  to 
align  it  with  the  President’s  priorities  as  the  full  President’s  budget  is  developed.  OSD  then  issues 
Program  Budget  Decisions  that  document  the  OSD  decisions  on  the  proposed  President’s  Budget. 
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Major  Budget  issues  will  arise  from  this  process  that  will  be  worked  at  the  Secretary  level.  Again 
the  content  of  the  Air  Force  program  will  be  adjusted  to  account  for  changes  that  may  occur  at  this 
level.  This  process  results  in  the  President’s  Budget  submission  to  Congress  in  January  the  year  of 
enactment. 


Congressional  Budget  Enactment 

Congressional  review  of  the  President’s  Budget  is  accomplished  by  a  range  of  committees. 
The  process  starts  with  the  Concurrent  Budget  Resolution  where  House  and  Senate  budget 
committees  agree  on  overall  spending  levels  for  different  components  of  the  federal  budget. 

The  Defense  Authorization  Bill  is  approved  in  the  house  by  two  house  subcommittees,  the 
House  National  Security  Committee,  and  the  full  House.  The  Senate  Defense  Authorization  Bill 
includes  reviews  by  six  senate  subcommittees,  the  Senate  Armed  Services  Committee,  and  full 
Senate.  A  conference  committee  works  out  the  differences  between  the  two  bills  which  is  then 
approved  again  by  the  House  and  Senate  and  signed  by  the  President.  This  authorizes  the  activities 
of  the  Department  of  Defense  and  is  similar  to  the  programming  phase  which  authorized  the  actions 
of  the  Department. 

The  Appropriations  Bill  is  reviewed  and  approved  in  the  house  by  two  subcommittees,  the 
House  Appropriations  Committee,  and  the  Full  House.  The  Senate  Appropriations  process  includes 
two  subcommittees ,  the  Senate  Appropriations  Committee  and  the  full  Senate.  A  conference 
committee  works  out  the  differences  and  it  is  passed  by  the  House  and  the  Senate  and  signed  by  the 
President. 


Budget  Execution 

Following  the  enactment  of  the  Defense  Budget,  modifications  to  the  Defense  program  must 
be  undertaken  to  comply  with  those  authorized  and  appropriated  by  Congress  and  the  budget 
authority  must  be  distributed  to  those  who  can  obligate  and  expend  it  to  buy  things  and  pay  people. 
The  Budget  Authority  is  passed  from  Congress  to  OMB  through  OSD,  to  the  services  and  then  to 
the  MAJCOMs.  Different  appropriation  categories  of  funds  have  different  Obligations  Availability 
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Periods  and  must  be  obligated  before  that  time.  Development  funding  can  be  obligated  for  up  to 
three  years.  Research  and  development  funding  can  be  obligated  for  up  to  two  years.  If  it  is  not 
obligated  to  a  contract  prior  to  this  time  it  is  returned  to  the  treasury.  The  services  go  to  great 
lengths  to  ensure  that  all  resources  are  used. 

The  result  of  the  Budgeting  systems  is  an  authorization  for  the  services  and  MAJCOMs  to 
spend  money  to  carry  out  their  programs.  The  budget  dictates  the  amount  that  can  be  spent  in  each 
area,  each  year,  and  on  which  tasks.  While  money  can  be  shifted  by  realigning  funds  within  a 
Program  Element  or  reprogrammed  between  program  elements  the  process  is  difficult,  time 
consuming,  and  accomplished  on  a  limited  bases.  The  Air  Force  can  approve  reprogramming  up  to 
$4  million  in  development  and  $10  million  in  production  funds.  Above  that  level  congressional 
approval  is  required. 

According  to  several  Pentagon  staff  personnel,  once  the  budget  is  passed  there  is  little  desire 
to  reopen  discussion  on  its  content.  Decisions  have  been  made  and  the  effort  turns  to  execution  of 
that  budget  and  the  development  of  the  next  years’  and  subsequent  budgets. 
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D.  Resource  Allocation  and  Planning,  Programming,  and  Budget 
Outcomes. 

The  modernization  planning  and  resource  allocation  process  are  precursors  to  the  start  of  the 
acquisition  process.  This  process  authorizes  a  development  activity  to  occur  and  determines  the 
resources  available  to  carry  it  out.  An  allocated  funding  profile  and  authorization  for  the  program 
element  are  required  to  proceed  past  a  Milestone  I  program  initiation  decision.  The  available 
resources  will  also  impact  the  program  development  planning.  The  smallest  unit  in  this  PPBS 
system  is  the  program  element.  The  program  element  is  overseen  by  a  program  element  monitor 
that  is  the  official  representative  in  the  Pentagon  of  that  program.  The  program  element  monitor  is 
the  person  in  the  Pentagon  most  familiar  with  an  individual  development  effort  and  the  effects  of 
the  funding  process  on  that  program. 


Appendix  3e 


Air  Force  Acquisition  Planning  Process 


A.  The  Acquisition  Process 

Acquisition  Program  -  a  directed,  funded  effort  designed  to  provide  a  new,  improved,  or 
continuing  weapon  system  capability  in  response  to  a  validated  operational  need. 

Source:  DoD  5000.1 


The  acquisition  process  is  designed  to  take  a  user’s  requirement  for  a  new  system  and 
design,  build,  and  field  that  system  in  a  cost  effective  manner.  This  process  is  detailed  in  the 
Federal  Acquisition  Regulations,  DoD  5000  series  regulations,  AFPD  63-1,  AFI 63-101,  and  the 
Air  Force  Acquisition  Model.  An  acquisition  program  officially  begins  at  Milestone  I  Program 
Initiation,  but  many  activities  are  accomplished  prior  to  the  Milestone  I  decision.  In  the  Pentagon 
the  execution  of  the  acquisition  process  is  managed  separately  from  the  planning  function  described 
earlier. 

The  key  players  in  the  execution  of  the  acquisition  efforts  are  the  program  offices,  the 
Program  Executive  Officers,  the  AFSARC,  the  Defense  Acquisition  Board,  and  several  levels  of 
integrated  product  teams  with  representation  from  the  various  functional  organizations.  The  people 
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and  organizations  involved  with  the  execution  of  the  acquisition  effort  overlap  significantly  with  the 
earlier  integrated  product  teams  in  the  resource  allocation  process  within  the  Pentagon.  These 
organizations  are  responsible  for  carrying  out  the  acquisition  efforts  within  the  allocated  resources 
provided  to  them. 

There  are  three  periods  within  any  acquisition  process;  the  acquisition  planning  period,  the 
contracting  period,  and  the  execution  period.  Acquisition  planning  includes  actions  such  as  the 
requirements  specification,  the  program  schedule  development,  the  acquisition  strategy 
development,  and  the  development  of  the  request  for  proposal.  Within  the  contracting  period  is 
contract  planning,  the  solicitation  process,  the  source  selection  process,  negotiation,  and  award 
processes.  The  actual  engineering  and  manufacturing  process  development  efforts  are 
accomplished  in  the  execution  phase  by  the  contractors  with  the  oversight  of  the  program  offices. 
These  contracting  cycles  may  be  repeated  within  a  development  project  as  the  selection  processes 
weans  the  number  of  contractors  competing  in  each  acquisition  phase  to  a  single  contractor  for  the 
EMD  or  production  phases. 

During  the  acquisition  process  there  are  three  primary  organizations  involved:  the  Pentagon, 
the  program  office,  and  the  defense  contractors.  Each  organization  plays  a  different  role  during  the 
different  periods.  The  acquisition  planning  and  contracting  efforts  are  led  by  the  program  office. 
Decisions  and  planning  approvals  for  major  ACAT I  and  II  programs  remains  at  the  Pentagon  level. 
For  smaller  ACAT  III  and  IV  programs  decisions  are  often  made  by  the  Center  commanders. 
Defense  contractors  provide  input  to  strategy,  develop  proposals,  and  carry  out  the  development 
after  contract  award.  The  activities  of  each  organization  will  be  discussed  in  each  acquisition 
periods. 


B.  Phases  and  Milestones 

DoD  5000  series  or  regulations  establishes  that  major  defense  acquisition  efforts  are  divided 
into  acquisition  phases  that  develop  the  program  into  a  full  system.  These  phases  are  intended  to 
take  a  system  from  an  idea  to  a  operationally  effective  fielded  system.  The  four  phases  that  are 
separated  by  milestone  decision  points  where  senior  leaders  review  the  progress  made  to  date  and 
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determine  whether  or  not  to  proceed  to  the  next  phase.  The  decision  whether  to  proceed  are  made 
by  either  the  Defense  Acquisition  Board  in  the  case  of  ACAT  ID  programs  or  at  the  service  level  for 
ACAT  1C  and  other  programs.  Different  activities  headed  by  the  groups  identified  earlier  plan 
different  roles  in  each  of  the  phases.  The  phases  are  defined  as: 


Pre-Milestone  0 
Phase  0 
Phase  I 
Phase  II 
Phase  III 
Phase  IV 


Determining  Mission  Need 
Concept  Exploration  and  Definition 
Demonstration  and  Validation 
Engineering  &  Manufacturing  Development 
Production  and  Deployment 
Operations  and  Support 


The  phases  are  seperated  by  milestones.  The  definition  for  the  milestones  are: 
Milestone  0  Concept  Studies  approval 

Milestone  1  Concept  Demonstration  Approval 

Milestone  2  Development  Approval 

Milestone  3  Production  Approval 

Milestone  4  Major  Modification  Approval 


The  pre-milestone  0  activities  are  not  officially  part  of  the  acquisition  process  as  the  acquisition 
process  as  the  acquisition  process  responds  to  plans  and  requests  from  the  Air  Force  planning  and 
requirements  processes  discussed  earlier.  An  actual  Acquisition  program  is  not  initiated  until 
Milestone  I. 
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ACQUISITION  MILESTONES  AND  PHASES 


Determination 
of  Mission  Need 


Phase  O 

Concept 
Exploration 
and  Definition 


Phase  1 

Demonstration 

and 

Validation 


Phase  II 

Engineering  and 
Manufacturing 
Development 


Phase  Pi 

Production 

and 

Deployment 


Phase  IV 

Operations 

and 

Support 


The  requirements  for  completing  each  of  the  phases  depends  considerable  upon  the  size  and 
importance  of  the  acquisition  effort.  Large  expensive  systems  receive  considerable  oversight  from 
the  services  and  from  DoD.  Smaller  programs  have  simplified  approval  processes  with  decisions 
made  at  lower  levels. 


Program  Size 

The  processes  used  to  run  the  defense  acquisition  system  depend  on  the  size  and  scope  of 
the  project.  Major  defense  acquisition  efforts  have  more  process  and  reporting  requirements  than 
smaller  programs  as  the  cost  and  visibility  of  potential  problems  are  significantly  larger.  Major 
Defense  Acquisition  Programs”  are  either  designated  as  by  OUSD(A&T)  or  are  estimated  to  cost 
more  that  $355  million  in  research  and  development  or  more  than  $2,135  billion  for  procurement  in 
FY  1996  dollars.  Programs  are  further  separated  mainly  by  development  and  production  costs  into 
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acquisition  categories  or  ACATs.  ACAT I  Programs  are  the  major  defense  acquisition  programs 
and  are  the  largest  programs  with  either  development  costs  over  $355  million  or  production  costs 
likely  to  exceed  $2,135  Billion.  ACAT  I  programs  are  further  designated  as  a  Defense  Acquisition 
Program  (ACAT  ID)  or  a  Component  Acquisition  program  (ACAT  IC)  indicating  whether  the 
Defense  Acquisition  Executive  (DAE)166  or  the  Component  Acquisition  Executive  (CAE)  has 
authority  over  the  program.  ACAT  II  programs  are  programs  below  the  threshold  for  ACAT  I 
programs  but  above  $140  Million  in  development  dollars  and  $645  Million  in  production  in  1996 
dollars.  ACAT  II  are  typically  run  by  the  Component  Acquisition  Executive  but  can  be  designated 
by  OUSD(A&T)  as  a  DoD  level  program  and  fall  the  same  reporting  requirements.  The  ACAT 
level  for  any  program  is  designated  by  the  Defense  Acquisition  Executive  OUSD(A&T). 

Category  III  and  IV  programs  are  considered  small  and  are  typically  run  by  the  product 
center  commanders  which  are  referred  to  as  Defense  Acquisition  Commanders  (DACs)167  which 
indicates  their  acquisition  authority.  The  separation  between  the  two  categories  is  ambiguous  but 
effects  some  of  the  reporting  requirements  and  is  an  effort  to  push  authority  for  the  program  to  a 
lower  level. 


166  Defense  Acquisition  Executive  (DAE)— The  principal  advisor  to  the  Secretary  of  Defense  (SECDEF)  on  all  matters 
pertaining  to  the  Department  of  Defense  acquisition  system  and  programs.  The  Under  Secretary  of  Defense 
(Acquisition)  is  the  DAE.  (AFPD  63-1) 

Defense  Acquisition  Board  (DAB)— The  Department  of  Defense  corporate  body  for  system  acquisition  providing 
advice  and  assistance  to  the  Secretary  of  Defense.  (DoD  Directive  5000.49) 

167 Designated  Acquisition  Commander  (DAC)-Tht  individual  who  functions  as  the  program  executive  officer  (PEO) 
on  programs  that  are  not  assigned  to  a  PEO.  The  commanders  of  product  divisions  and  air  logistic  centers  act  in  this 
capacity.  DACs,  like  PEOs,  are  accountable  to  the  Air  Force  Acquisition  Executive  (AFAE).  (AFPD  63-1) 
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Acquisition  Development  Production  Decision 

Categories  Costs  Costs  Authority 

ACATI  $355  M 

ACAT  ID 
ACAT IC 


DAE 

CAE 


ACAT  II  $140  M 


$  645  M  CAE 


ACAT  III 
ACAT  IV 


Less  than  S140M  Less  than  $645  M  CAE,  DAC 

Less  than  ACAT  III  Less  Than  ACAT  III  DAC  or  lower 


Automated  Information  Systems 


Development 

Cost 


Life  Cycle 
Costs 


Maximum  Decision 

Cost  per  year  Authority 


ACAT IM 
ACAT  III 
ACAT  IV 


$120  M 

Less  than  $120M 
Less  than  ACAT  III 


$360  M  or 

Less  than  $360  M 

Less  Than  ACAT  III 


$30  M  ASD(C3I) 

$30M  CAE,  DAC 

DAC  or  lower 


Table  26  Acquisition  Categories,  Dollar  Amounts  and  Decision  Authorities 

Numbers  listed  in  1996  dollars. 


The  size  of  an  acquisition  effort  effects  the  processes  used  to  develop  and  approve  the 
program.  Larger  programs  because  of  their  larger  dollar  value  receive  considerably  more  attention. 
The  underlying  process  is  the  same  for  large  and  small  programs.  All  include  aspects  of  the 
dveloping  requirement,  identifying  resources,  and  accomplishing  the  acquisition  process. 


C.  Acquisition  Planning 

During  the  acquisition  planning  period  the  entire  development  effort  is  defined,  scheduled, 
and  structured.  This  process  takes  the  requirements  from  the  user’s  Operational  Requirements 
Document  and  turns  them  into  executable  plans  within  the  identified  resources  that  can  be  used  to 
run  an  acquisition  effort  from  program  start  to  final  deployment.  These  plans  cover  areas  such  as 
acquisition  procedures,  integrated  logistics  support,  test  and  evaluation,  and  program  goals, 
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specifications,  and  requirements.  This  effort  is  accomplished  by  the  newly  formed  program  office 
or  by  a  team  within  an  existing  program  office.  The  program  office  will  typically  begin  a  market 
survey  of  potential  contractors  to  learn  about  capabilities  and  technologies  that  may  be  used  in  the 
acquisition  effort. 

Much  of  this  planning  process  is  accomplished  prior  to  the  Milestone  I  program  initiation. 
The  scope  and  the  extent  of  the  acquisition  planning  phase  are  captured  in  the  reports  and 
information  that  is  required  for  the  Milestone  I  decision.  Much  of  the  acquisition  planning  period  is 
spent  developing  the  plans  and  capturing  the  information  for  the  reports  required  fro  Milestone  I. 
these  reports  are  indicated  in  the  Table  below: 


Acquisition  Strategy  Report 

Acquisition  Program  Baseline 

Affordability  Assessment 

FYDP  Funding  Profile 

Analysis  of  Alternatives 

Component  Cost  Analysis 

Cost  Analysis  Requirements  Description 


Description  of  Data  Requirements 
Exit  Criteria 

Independent  Estimate  of  Life  Cycle  Costs 
Operational  Requirements  Document 
Program  Office  Life  Cycle  Cost  Estimate 
System  Threat  Assessment  Report 
Test  and  Evaluation  Master  Plan 


Table  A3D-27:  Information  Required  for  Milestone  I  Decisions  for  Major  Defense 
Acquisition  Programs 


C.l.  Establish  Program  Office 

For  new  weapon  systems  the  pre-Milestone  0  and  early  Phase  0  activities  are  typically 
supported  by  the  Plans  and  Programs  (XR)  organizations  at  the  various  product  centers.  When  it 
becomes  clear  that  a  program  will  be  established,  a  program  office  is  established  and  a  program 
manager  is  assigned.  According  to  the  DSMC  wall  chart,  the  program  manager  is  to  be  assigned 
within  6  months  of  a  successful  Milestone  I  decision.  For  modifications  to  exisiting  systems  a 
program  office  may  already  exist,  and  a  new  team  will  be  formed  from  within  the  existing  office. 
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C.2.  Acquisition  Strategy 

The  acquisition  strategy  is  a  roadmap  for  the  program  execution  from  initiation  to  post¬ 
production  support.  The  acquisition  strategy  covers  the  essential  elements  of  the  program.  Art  a 
minimum,  it  covers  the  objectives  for  the  product,  risk  management,  cost  objectives,  contracting 
approach,  management  approach,  environmental  considerations,  and  sources  of  support.  The 
acquisition  strategy  includes  the  events  that  govern  the  management  of  the  program. 

The  acquisition  strategy  is  developed  by  the  program  manager  for  program  initiation  and 
approved  at  the  Milestone  I  review.  The  acquisition  strategy  is  updated  and  changed  as  needed  to 
reflect  programmatic  and  system  changes.  Changes  are  coordinated  with  the  working-level  IPTs  (to 
be  described  later)  and  approved  by  the  milestone  decision  authority  prior  to  release  of  the  RFP. 

The  acquisition  strategy  identified  the  sources  that  are  available  to  develop  the  intended 
system  taking  into  account  commercial  equivalent  systems,  industrial  capability  and  socio¬ 
economic  preference  factors.  In  the  area  of  risk  management,  key  risk  drivers  must  be  identified 
and  tracked.  Risk  includes  cost,  schedule,  and  performance  risks.  Strategies  to  minimize  risk  are 
identified. 

The  acquisition  strategy  is  reviewed  by  a  panel  of  senior  acquisition  officers  in  an 
acquisition  strategy  panel.  Here  constructive  feedback  is  provided  to  the  program  manager. 

The  acquisition  strategy  is  captured  in  the  Acquisition  Strategy  Report168  which  is  one  of  the 
required  documents  for  a  Milestone  I  decision.  It  is  also  included  in  a  more  detailed  acquisition 
plan  maintained  at  the  program  office. 


168  Acquisition  Strategy  Report  (ASR)— Describes  the  acquisition  approach,  to  include  streamlining, 
sources,  competition,  and  contract  types,  throughout  the  period  from  the  beginning  of  Phase  I, 
Demonstration  and  Validation,  through  the  end  of  production.  (DoD  Instruction  5000.2/ Air  Force 
Supplement  1) 
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The  acquisition  strategy  development  is  the  first  part  of  the  contracting  process.  Much  of 
the  contracting  procedures  are  dictated  by  the  Federal  Acquisition  Regulations  and  other  federal 
laws.  The  contracting  officers  are  responsible  for  ensuring  that  all  laws  and  regulations  are  followed 
in  the  contracting  activities.  Acquisition  strategy  development  is  documented  in  the  Acquisiton 
Strategy  Report. 
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The  Acquisition  Strategy  Report  Content 


Intelligence  Support 

Requirements  Evolution 

Analysis  of  Alternatives 

Affordability 

Supportability 

Program  Structure 

Program  Goals  and  Thresholds 

Acquisition  Program  baselines 

Acquisition  Phase  Exit  Criteria 

Acquisition  Strategy 

Potential  Sources 

Risk  Management 

Cost  as  an  Independent  Variable 

Objectives 

Contract  Approach 


Competition 

Best  Practices 

Cost  Tracking  Approach 

Management  Approach 
Streamlining 

Joint  Program  Management 
Program  Executive 
Responsibility 

Oversight 
Sources  of  Support 
Warranties 

Test  and  Evaluation 

Test  and  Evaluation  Strategy 
DT&E 


Table  28:  Outline  of  an  Acquisition  Strategy  Report 


C.3.  Acquisition  Program  Baseline 

The  Acquisition  Program  Baseline  (APB)169  is  formal  agreement  between  the  key  players  in 
an  acquisition  effort.  It  represents  an  understanding  of  the  key  aspects  of  the  program  which  are 
critical  to  its  success.  Every  acquisition  program  is  required  to  have  an  acquisition  program 
baseline  (APB).  It  is  intended  to  enhance  the  program  stability  and  provide  a  reference  with  which 
to  judge  the  progress  and  status  of  a  program. 


169  Acquisition  Program  Baseline  (APB)— A  succinct  document  that  consists  of  cost,  schedule,  and 
performance  (including  support)  parameters  that  establishes  the  commitment  between  the  program 
manager  and  the  applicable  Milestone  Decision  Authority.  (AFPD  63-1) 
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The  APB  documents  the  cost,  schedule  and  performance  objectives  and  thresholds  for  that 
program.  The  APB  is  limited  to  only  the  most  important  parameters  of  performance,  schedule,  and 
costs.  The  performance  section  identifies  both  objectives  and  threshold  values  for  key  performance 
parameters  that  are  identified  in  the  Operational  requirements  and  maintainability  parameters.  The 
threshold  values  are  the  minimum  acceptable  parameters  which  if  not  met  require  a  review  by  the 
milestone  decision  authority  to  determine  whether  or  not  to  proceed  with  the  program.  The  “trade 
space”  with  which  the  program  manager  can  operate  without  MDA  approval  is  the  difference 
between  the  threshold  and  the  objective  parameters.  The  schedule  section  of  the  APB  details  the 
significant  expected  events  each  year.  Schedule  milestone  examples  may  be  milestone  decision 
points,  major  design  reviews,  first  delivery,  major  tests,  and  initial  and  full  operational  capability. 
The  cost  section  of  the  APB  details  the  program  baseline  costs  by  appropriation  type  and  year.  It 
also  identifies  unit  procurement  costs  and  total  procurement  quantities. 

Baselines  are  approved  at  each  milestone  and  cover  the  remainder  of  the  program.  The 
baselines  are  refined  at  each  successive  review.  The  baseline  approved  at  Milestone  I  is  known  as 
the  concept  baseline.  The  development  baseline  approved  at  milestone  II  is  further  refined,  and  the 
production  baseline  approved  at  milestone  III.  This  successive  refining  of  the  program  baselines  is 
intended  to  prevent  premature  commitment  to  a  system  specific  solution. 

The  breach  of  a  program  baseline  brings  on  high  level  reviews  and  increased  scrutiny  for  a 
program.  An  APB  breach  can  be  brought  on  by  an  inability  to  meet  performance  thresholds;  a 
schedule  slip  of  6  months;  cost  growth  exceeding  15%  in  RDT&E,  MILCON,  or  unit  procurement 
price;  or  a  growth  of  5%  in  total  procurement  price.  Program  breaches  are  reported  with  a  Program 
Deviation  Report  or  typically  by  a  baseline  change  request  prior  to  the  breach. 


C.4.  Program  Plans 

Program  plans  are  detailed  descriptions  of  the  activities  required  to  develop  and  implement 
the  acquisition  strategy.  These  program  plans  are  controlled  by  the  program  office  with  oversight 
by  the  PEO  or  the  center  commanders.  These  plans  are  detailed  roadmaps  for  the  entire  acquisition 
and  development  effort  leading  from  Milestone  1  to  the  end  of  production.  The  plans  are  intend  to 
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ensure  all  issues  have  been  addressed  prior  to  the  acquisition  process  initiation.  The  plans  are 
coordinated  at  the  functional  representatives  at  the  command  level  by  the  program  offices  and 
Pentagon  levels  by  the  working  and  overarching  IPTs.  Here  the  interested  functional  parties  can 
provide  input  and  flag  concerns  that  are  to  be  addressed  prior  to  the  milestone  review  or  run  the  risk 
a  of  “non-concur”  from  a  Pentagon  functional  area  on  their  program  approval  during  the  milestone 
review  process.  Officially  these  plans  are  not  part  of  the  milestone  documentation  process  but  they 
are  summarized  or  attached  to  the  integrated  program  summary  that  initiates  the  milestone  review 
process.  Varying  levels  of  detail  for  each  plan  are  included  at  each  of  the  milestone  reviews  and 
progressively  become  more  detailed  over  time. 
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Various  Program  Plans 


Acquisition  Plan  -  The  acquisition  plan  is  maintained  by  the  procuring  contracting  officer  and 
details  the  steps  required  to  procure  the  system  and  its  support  through  the  contracting  plan.  This  a 
plan  is  vetted  through  an  acquisition  strategy  panel.  The  acquisition  plan  will  be  discussed  further 
in  the  contracting  process  below. 

Source  Selection  Plan  -  The  source  selection  plan  is  developed  prior  to  RFP  release  and  details  the 
procedures  and  evaluation  criteria  that  will  be  used  to  make  the  source  selection  decision.  This  will 
be  described  in  more  detail  in  the  contracting  section  below. 

System  Engineering  Plans  -  The  Government  Work  Breakdown  Structure,  System  Engineering 
Master  Plan,  and  the  System  Engineering  Master  Schedule  are  intended  to  provide  support  to  the 
acquisition  planning  process  and  to  identify  the  key  system  engineering  items  in  the  development 
process.  The  WBS,  SEMP  and  SEMS  are  described  in  the  System  Engineering  Military  Standard 
499B. 

Work  Breakdown  Structure  (WBS)  -  The  government  develops  a  work  breakdown  structure  for 
the  development  effort  based  on  the  currently  defined  specification  tree  for  the  system.  This  work 
breakdown  structure  is  used  to  develop  an  organizational  structure  for  both  the  government  and  the 
contractor.  It  is  also  used  to  structure  key  source  selection  criteria.  The  contractor  is  expected  to 
extend  the  work  breakdown  structure  as  part  of  their  proposal  to  a  level  necessary  to  complete  the 
contractual  requirements. 

Systems  Engineering  Master  Plan  (SEMP)  -  A  System  Engineering  Master  Plan  is  developed  by 
the  program  office  for  planning  purposes  in  its  acquisition  effort  and  addresses: 

-  Life  cycle  roadmap  for  key  system  engineering  activities  and  who  (government  or 
contractor)  is  responsible  for  their  accomplishment. 

-  The  government  organizational  structure  and  responsibilities  for  overseeing  the 
contractor’s  technical  efforts. 

-  Plans  and  criteria  for  transitioning  key  product  and  process  technologies 


458  Reducing  DoD  Product  Development  Time:  The  Role  of  the  Schedule  Development  Process 


-  Key  trade  studies,  scope  of  system  effectiveness  assessment,  priority  of  measures  of 
effectiveness,  technical  risk  management  plans,  critical  technical  parameters,  and  tracking 
requirements  for  those  parameters. 


Systems  Engineering  Master  Schedule  (SEMS)  -  The  government  SEMS  provides  top-level 
events,  accomplishments,  and  accomplishment  criteria  for  the  entire  program. 

Together  the  government  SEMP  and  the  SEMS  provide  the  detailed  information  for  the  next 
acquisition  phase  and  engineering  effort  to  identify  the  specific  events,  accomplishments,  and 
criteria  necessary  to  satisfy  the  planned  and  required  technical  exit  criteria.  The  Government  SEMP 
and  the  SEMS  are  often  provided  in  part  or  total  as  part  of  the  Request  for  Proposal  Package.  When 
it  is  included  in  the  RFP  it  is  intended  for  “guidance  only”  .'70  The  contractors  are  required  to  submit 
their  System  Engineering  Master  Plans  and  the  System  Engineering  Master  Schedule  with  their 
proposals.  The  contractors  also  develop  the  System  Engineering  Detailed  Schedule  that  shows  how 
the  contractor  is  planning  their  work  to  meet  the  Master  Schedule.  These  represent  the  contractor’s 
specific  plans  to  carry  out  the  development  effort.  The  contractor  SEMP  is  approved  for  compliance 
by  the  government.  The  negotiated  contractor  SEMS  will  be  placed  on  contract  and  requires  a 
contract  modification  to  change.  These  documents,  along  with  the  work  breakdown  structure 
(WBS)  become  the  critical  management  control  documents  following  contract  award. 


Integrated  Logistics  Support  Plan  (ILSP)  -  The  Integrated  Logistics  Support  Plan  is  the 
government’s  formal  planning  document  for  logistics  support.  The  document  is  prepared  by  the 
program  office  deputy  program  manager  for  logistics.  It  is  the  result  of  the  Integrated  Logistics 
Support  Analysis  (LSA)  which  is  covered  by  Mil-Stdl338-1  A  and  2A/2B.  Integrated  logistics 
analysis  is  a  key  determinate  in  the  lifecycle  cost  estimates  used  at  the  Milestone  I  decision.  LSA 
includes  10  elements  that  are  included  in  the  integrated  logistics  support  plan.  They  include: 
maintenance  planning;  manpower  and  personnel;  supply  support;  support  equipment;  technical  data; 


170  Mil-Std-499B  para  6.4.1  Application  Guidance. 
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training  and  training  support;  computer  resources  support;  facilities;  packaging,  handling,  storage, 
and  transportation;  and  design  interfaces.  Each  element  listed  has  own  set  of  processes,  procedures, 
and  techniques  used  to  satisfy  the  different  requirements  of  each  element.  The  process  is  intended 
to  identify  cost  drivers  following  system  deployment  and  identify  and  plan  for  the  resources 
required  to  support  it.  The  Integrated  Logistics  Support  Plan  is  developed  prior  to  the  milestone  I 
review  and  is  updated  and  reviewed  at  the  each  Milestone  review. 


Test  and  Evaluation  Master  Plan  -  The  Test  and  Evaluation  Master  Plan171  details  the 
entire  testing  program  to  be  used  at  the  start  of  the  acquisition  process  to  test  the  system,  including 
the  Development  Testing  and  Evaluation  (DT&E)  and  the  operational  testing  (OT&E)  prior  to  a  full 
rate  production  decision.  The  TEMP  identifies  the  approaches,  the  resources  required,  and  the 
facilities  required.  The  development  of  this  plan  is  accomplished  by  the  Test  Planning  Working 
Group  an  integrated  product  team  of  program  office,  user,  AFOTEC,  contractors  (once  selected), 
and  test  center  people.  The  TEMP  is  a  requirement  for  a  milestone  I  decision.  It  includes  the 
system  information,  a  program  summary,  a  detailed  Development  Test  and  Evaluation  plan  outline, 
a  detailed  Operational  Test  and  Evaluation  plan  outline,  and  a  detailed  resource  summary.  The 
DT&E  and  OT&E  plans  describe  how  the  critical  operating  issues,  measures  of  performance  and 
measures  of  effectiveness  drawn  from  the  operational  requirements  document  and  requirements 
correlation  matrix  will  be  evaluated.  The  resource  summary  identifies  the  resources  and  schedule 
for  the  test  program  including  test  articles,  instrumentation,  test  facilities,  support  equipment,  and 
funding,  manpower,  and  personnel  requirements.  To  obtain  test  support  a  program  manager  can  use 
the  existing  resources  within  the  DoD  test  community,  go  through  the  Test  Investment  Planning  and 
Programming  Process,  send  funds  to  the  test  facility  for  an  upgrade,  or  pay  industry  to  develop  the 
test  equipment.  The  last  two  require  additional  funding  from  the  program  office  and  are  seen  as  the 
last  option.  The  TEMP  is  updated  and  refined  at  each  milestone  review. 


171  Test  and  Evaluation  Master  Plan  (TEMP)— The  basic  planning  document  for  all  test  and  evaluation  related  to  a 
particular  system  acquisition  which  is  used  in  planning,  reviewing,  and  approving  test  and  evaluation.  (DoD 
Instruction  5000.2) 
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Human  Systems  Integration  (HSI)  Plan  or  Integrated  Manpower,  Personnel,  and 
Comprehensive  Training  and  Safety  (IMPACT)  Program  Plan  (IPP)  ~  The  Human  System 
Integration  Plan  documents  the  manpower,  force  structure,  and  training  requirements  for  a  new 
program.  This  plan  is  responsive  to  the  manpower  communities  so  they  can  ensure  the  necessary 
resources  are  available  for  the  deployment  of  the  system.  The  Air  Force  implements  HSI  through 
the  Integrated  Manpower,  Personnel,  and  Comprehensive  Training  and  Safety  (IMPACTS)  Program 
Plan  (IPP)  and  the  Preliminary  IMPACTS  Program  Plan.  It  is  also  the  background  analysis  and 
justification  for  the  Manpower  Estimate  Report  (MER).  The  preliminary  IMPACTS  Program  Plan 
is  reviewed  prior  to  Milestone  I 


Intelligence  Support  Plan  (ISP)  -  The  intelligence  support  plan  documents  the  intelligence 
resources  required  for  the  implementation  of  the  new  system.  It  identifies  the  resources  required 
and  the  level,  timeliness,  and  accuracy  of  the  information  required.  This  plan  is  coordinated  with 
the  intelligence  community  to  ensure  they  are  capable  and  staffed  to  support  the  new  system. 

System  Threat  Assessment  Report  (STAR)--The  STAR  is  an  intelligence  document  that  serves  as 
the  single  reference  for  threat  information  regarding  a  weapon  system  acquisition  program.  It 
describes  the  lethal  and  non-lethal  threats  against  the  proposed  US  system  and  the  threat 
environment  in  which  the  system  will  operate.  (AFM  11-1) 


C.5.  Cost  and  Manpower  Estimates 

Prior  to  program  initiation  detailed  life  cycle  cost  estimates  are  required.  The  program  office 
develops  detailed  estimates  that  are  “  comprehensive  in  character,  identify  all  elements  of  cost  that 
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would  be  effected  by  a  decision  proceed.”  m  The  basis  for  these  estimates  are  the  program  plans 
developed  by  the  program  office  and  analysis  from  the  functional  areas.  The  Cost  Analysis 
Requirements  Document  defines  the  factors  to  be  analyzed  for  the  lifecycle  cost  of  a  system  and 
provides  a  structure  for  the  three  cost  estimates:  the  program  office  cost  estimate,  the  component 
cost  estimate,  and  the  independent  cost  estimate. 


Cost  Analysis  Requirements  Document  (CARD)  -  The  Cost  Analysis  Requirements  Document  is 
the  description  of  the  salient  features  of  the  acquisition  program  and  the  system  proposed  for 
development  for  the  basis  of  the  life-cycle  cost  estimates.  Life  Cycle  Costs  include  all  the 
procurement  and  fielding  costs  plus  the  operations  and  support,  military  personnel,  modification 
and  disposal  costs.  Having  a  common  basis  between  estimates  allows  for  comparisons.  This 
document  is  developed  by  the  services  and  forwarded  to  the  program  office,  the  service  cost 
analysis  agency,  and  the  DoD  Cost  Analysis  Improvement  Group  (CAIG)  at  least  180  days  prior  to 
the  Overaching  IPT  meeting  prior  to  a  Defense  Acquisition  Board  review  or  a  component  review. 
This  is  to  allow  adequate  time  for  analysis  of  the  expected  lifecycle  costs  of  the  proposed  system. 
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Figure  16:  Definition  of  Life-Cycle  Costs173 
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Program  Office  Cost  Estimate  -  The  program  office  cost  estimate  is  the  life  cycle  cost  estimate  for 
the  proposed  system.  The  program  office  estimate  is  developed  by  the  financial  management 
organization  within  the  program  office. 


Component  Cost  Analysis  Estimate  -  The  component  cost  analysis  estimate  is  developed  in  the 
Air  Force  by  the  Air  Force  Cost  Analysis  Agency  (AFCAA)  under  SAF/FM.  It  is  an  analysis  of 
program  cost,  as  well 


Independent  Cost  Estimate  -  The  Independent  Cost  Estimate  is  developed  by  OSD  staff  to  provide 
decision  makers  with  estimate  from  outside  an  advocating  service.  This  estimate  is  developed  by 
the  Cost  Analysis  Improvement  Group  (CAIG)  that  was  established  solely  for  this  purpose. 

I 

Cost  As  Independent  Variable  (CAIV)  Objectives  - 

A  recent  initiative  known  as  Cost  as  an  Independent  Variable  (CAIV)  is  designed  to  include 
life  cycle  cost  as  a  critical  factor  in  making  performance  tradeoffs.  The  intention  is  to  establish 
affordability  goals  for  the  program  and  treat  the  cost  not  as  an  outcome  of  the  development  process 
but  to  use  cost  as  a  driver  of  the  performance  tradeoffs.  The  CAIV  objectives  indicate  the  cost 
objectives  and  thresholds  for  the  program.  The  CAIV  objectives  are  also  reviewed  at  the  working 
and  overarching  IPT  levels. 


Following  the  completion  of  the  program  plans  and  the  other  Phase  0  activities,  the  final 
step  in  preparation  for  a  milestone  decision  is  the  development  of  the  integrated  program  summary. 


Integrated  Program  Summary  -  The  Integrated  Program  Summary  is  the  summation  report  of  the 
previous  phase  with  all  the  pertinent  information  to  justify  a  decision  to  proceed  to  the  next  phase.  It 
is  the  decision  document  that  once  approved  by  the  Defense  Acquisition  Board  allows  the  program 
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to  proceed.  It  includes  summaries  of  the  Acquisition  Strategy,  the  status  of  the  program,  the  threat, 
the  alternatives  explored,  the  risks  identified,  the  cost  drivers,  and  summaries  of  the  various 
program  plans.  The  completion  and  submission  of  the  Integrated  Program  Summary  initiates  the 
milestone  review  process. 


D.  Acquisition  Milestone  Approval  Process 

The  milestone  approval  process  is  the  formal  process  used  to  allow  a  program  to  proceed  to  the  next 
phase  of  activities.  It  is  designed  to  ensure  that  all  necessary  issues  are  addressed  prior  to  the 
approval  of  the  program  to  proceed.  The  process  has  undergone  significant  changes  in  the  last 
decade  and  the  average  approval  process  once  all  the  issues  have  been  resolved  has  been  cut 
dramatically  in  the  last  several  years.  The  development  of  the  Working-level  IPT  and  Overaching 
IPT  structure  has  increased  coordination  on  DAB  packages  prior  to  the  actual  meeting  of  the 
Defense  Acquisition  Board  to  the  point  that  all  the  issues  are  expected  to  be  worked  by  the 
respective  level  IPT  before  the  DAB  meets. 
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Figure  17:  Milestone  Review  Process 
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This  has  allowed  the  actual  cancellation  of  the  DAB  meeting  and  the  use  of  what  is  referred  to  as  a 
“Paper  DAB” .  Here  the  principle  functional  Deputy  Chiefs  of  Staff  sign  off  on  a  pre-coordinated 
package  that  is  forwarded  to  the  Milestone  Decision  Authority  for  final  approval  that  serves  as  the 
official  approval  decision.  Prior  to  the  development  of  the  Working  and  Overaching  IPTs,  the  DAB 
was  the  place  where  issues  were  raised  about  particular  programs  and  the  issues  were  worked 
following  the  meeting  of  the  DAB.  This  was  by  its  very  nature  was  late  in  the  development  of 
program  plans  and  often  caused  significant  delays  in  the  approval  process.  The  working  level  IPT 
allows  for  earlier  functional  input  and  consultation  into  the  program  plans  and  allows  the  issues  to 
be  worked  prior  to  the  submission  of  the  Integrated  Program  Summary.  These  reforms  plus  the 
shortening  of  the  acquisition  chain  of  command  due  to  the  Packard  Commission  has  dramatically 
shortened  the  time  required  for  program  decisions.174  This  does  not  mean  that  the  time  required  to 
work  the  issues  prior  to  the  meeting  of  the  DAB  has  been  reduced  but  it  does  mean  that  delays 
during  the  approval  process  because  of  issues  rising  out  of  an  unsuccessful  DAB  are  reduced  or 
eliminated. 

For  a  point  of  reference  of  the  size  of  the  Overaching  and  Working  Level  IPTs,  the  F-22  has 
29  organizations  from  OSD,  JCS  and  the  Air  Force  represented  on  its  OIPT  and  60  different 
organizations  represented  on  its  WIPT.  Smaller  programs  will  have  fewer  groups  interested  and 
involved  in  the  IPTs.175  The  information  required  for  the  milestone  decisions  at  each  milestone 
review  is  listed  in  the  tables  below. 

The  outcome  of  the  milestone  decision  is  a  signed  Acquisition  Decision  Memorandum  that 
authorizes  the  next  phase  of  activities.  It  also  establishes  the  exit  criteria  that  must  be  met  in  that 
phase  before  the  program  can  proceed  to  later  phases.  For  a  Milestone  I  decision  this  authorized 
the  start  of  an  acquisition  program.  With  this  approval  the  official  contracting  process  can  be 
accomplished  to  select  a  contractor  to  develop  the  desired  system. 


174  The  Packard  Commission  and  its  implementing  laws  shortened  the  acquisiton  chain  of  commnd  by  establishing  the 
PEO/DAC  structure  that  reports  directly  to  the  SAE.  This  removed  Air  Force  System  Command  and  much  of  Air  Staff 
from  the  chain  of  command. 

175  General  Hawley’s  Briefing  to  the  AQ/XO  offsite. 
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Figure  18:  Hierarchy  of  Integrated  Product  Teams  for  Acquisition  Execution.  Taken  from 
General  Hawley’s  Briefing  for  AQ/XO  Offsite. 
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Information  for  Milestone  Reviews  -  ACAT I  and  ACAT IA  Programs 


Information 
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MDA's  for  ACAT  II  &  III  programs  have  wide  latitude  and  broad  authority 


over  the  content  and  format  of  many  (but  not  all)  of  these  information  elements 


1.  MS  0  for  AIS  programs;  MS  I  for  others.  May  be  required  by  MDA  for  MS  n/m. 

2.  May  be  included  in  acquisition  strategy. 

3.  Normally  not  required  for  AIS  programs. 

4.  Covered  ACAT  II  &  product  improvements  to  covered  systems. 

5.  ACAT  II  only. 

6.  Programs  on  OSD  T&E  Oversight  List  and  others  designated  by  MDA. 


Figure  19:  Information  Required  for  Milestone  Reviews  for  ACAT  I  and  ACAT  II  &  III 
programs 


Appendix  If 


Air  Force  Contracting  Process 


The  contracting  process  is  a  very  significant  part  of  the  development  process.  Often  the 
contracting  process  is  not  viewed  separately  from  the  development  process  but  the  development 
process  is  larger  and  more  encompassing  than  the  contracting  process  alone.  The  basic  contracting 
process  is  dictated  by  the  Federal  Acquisitions  Regulations  and  apply  across  all  federal  agencies. 
The  contracting  process  in  the  Air  Force  is  covered  by  the  DoD  and  Air  Force  supplements  to  the 
Federal  Acquisition  Regulation.  These  regulations  and  the  associated  policy  directives, 
instructions,  and  court  findings  currently  dictate  the  procedures  to  be  followed  in  the  acquisition 
process.  The  Federal,  DoD,  and  Air  Force  contracting  process  are  designed  to  ensure  integrity  of 
the  processes,  that  fair  and  open  competition  prevails,  and  the  applicable  laws  and  socioeconomic 
policies  are  carried  out. 

The  two  primary  purposes  of  the  contracting  and  source  selection  processes  are  to  select  a 
contractor  and  to  agree  on  terms  and  conditions  for  the  contract.  The  FAR  outlines  two  primary 
processes  for  selecting  and  awarding  contracts:  sealed  bidding  and  the  competitive  proposal  process 
which  is  also  known  as  negotiation.  Sealed  bidding  is  used  when  price  is  the  only  evaluation  factor. 
The  sealed  bidding  process  is  described  by  FAR  Part  14  and  does  not  allow  any  form  of  discussion 
or  negotiation  prior  to  the  awarding  of  the  contract.  The  competitive  proposal  process  is  used  for 
the  development  of  nearly  all  DoD  systems.  In  the  competitive  proposal  process  of  awarding 
contracts,  discussions  and  bargaining  are  permitted.  The  competitive  negotiation  process  is 
described  in  the  Federal  Acquisition  Regulation  Part  15  Contracting  by  Negotiation  which  provides 
a  description  of  the  processes  required  to  be  followed.  According  to  the  FAR,  the  negotiation 
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procedure  includes  receipt  of  proposals  from  offerors,  permits  bargaining,  and  usually  affords 
offerors  an  opportunity  to  revise  their  offers  before  award  of  a  contract.  The  FAR  details  the 
structure  of  the  Request  for  Proposals  (15.4)  and  the  source  selection  process  (15.6).  It  also 
established  the  guidelines  for  price  negotiations,  profit,  and  the  notifications  and  debriefing 
processes  (15.8,9,10). 

The  general  requirement  for  the  negotiation  contracting  process  is  full  and  open  competition 
which  is  also  dictated  by  the  Competition  In  Contracting  Act  (CICA).  For  other  then  full  and  open 
competition  a  justification  and  approval  (J&A)  is  required  and  is  approved  only  in  limited 
conditions.  Non-competitive  procurement  or  the  sole  source  process  is  used  primarily  for 
upgrading  existing  systems  that  are  in  production  or  following  a  previous  selection  process  for 
follow-on  contracts.. 

The  contracting  process  can  be  separated  into  three  logical  periods:  the  contract  planning 
period  leading  up  to  the  RFP  release,  the  solicitation  and  source  selection  period  leading  to  contract 
award,  and  the  execution  period  when  the  work  is  carried  out.  This  planning  process  overlaps 
significantly  and  is  often  indistinguishable  from  some  of  the  early  overall  acquisition  planning 
process. 
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Figure  8:  Generic  Contracting  Process  for  Competitive  and  Non-competitive  Contracts 


A.  Contract  Planning  -  Presolicitation  Phase 

There  are  many  steps  in  the  contract  planning  phase.  All  are  required  to  get  a  development 
program  on  contract.  This  process  culminates  in  the  development  of  the  final  RFP  that  is  ready  to 
be  released  to  industry. 
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Market  Research  -  The  first  step  in  the  contracting  process  is  to  accomplish  market 
research  and  determine  how  many  companies  could  accomplish  the  tasks  requested.  This  is  done 
by  a  sources  sought  synopsis  in  the  commerce  business  daily.  Responding  companies  are  sent  a 
request  for  information  which  is  then  used  to  determine  if  they  are  potential  sources.  Often  site 
visits  will  be  accomplished  and  meeting  are  held  to  gain  information  from  the  companies  as  to  their 
capabilities.  The  purpose  is  to  identify  any  company  interested  in  bidding  on  the  contract  and 
determine  if  they  have  a  reasonable  chance  of  accomplishing  the  tasks  for  the  development  of  the 
acquisition  strategy.  During  this  period  the  companies  will  often  try  to  present  their  approaches  to 
the  problems  in  an  attempt  to  influence  the  specifications  and  requirements  of  the  system  in  the 
RFP. 

Having  identified  potential  sources  for  the  effort  a  strategy  can  be  identified  to  select  a 
contractor  to  design  and  build  the  equipment  that  is  desired. 

Determine  Program  Schedule  -  During  this  period  the  schedule  for  the  acquisition 
program  is  developed.  This  has  typically  included  major  milestones  and  reviews  as  established  in 
the  System  Engineering  Master  Schedule  or  the  Intergrated  Program  Master  Schedule. 

Write  Specification/Statement  of  Work/Statement  of  Objective  -  The  details  of  what  the 
government  wants  to  buy  or  what  the  government  wants  the  system  to  do  must  be  specified  to 
provide  guidance  to  the  contractors  as  to  the  desires  of  the  goverenment.  This  is  done  through  the 
requirements,  specifications,  and  the  statement  of  work/objectives  included  in  the  RFP.  The 
industry  is  consulted  as  to  their  input  into  the  requirements  through  formal  and  informal  meetings 
between  the  program  office  personnel  and  the  various  interested  contractors.  Contractor  support 
and  FFRDC  engineers  (such  as  Mitre  or  Aerospace)  typically  assist  in  the  writing  of  the 
specificatons  and  requirements. 

Cost  Estimate  -  The  costs  are  estimated  for  the  specifcations  and  requirements  identified  to 
in  the  RFP  to  ensure  the  expected  cost  of  the  contract  is  within  the  resources  available. 

Acquisition  Plan  -  The  Acquisition  Plan  is  developed  by  the  program  office  and  maintained 
primarily  by  the  program  office’s  contracting  officer.  This  document  describes  the  process  that  will 
be  used  to  get  the  effort  on  contract.  It  outlines  both  the  strategy  to  be  used,  the  contracting  process 
to  be  used,  and  the  specific  steps  that  are  required  to  be  taken.  It  also  lays  out  the  timeline  for  the 
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steps,  to  take.  It  is  the  plan  that  the  program  office  can  follow  to  ensure  that  all  steps  have  been 
accomplished. 

The  review  of  the  proposed  acquisition  strategy  is  done  by  an  Acquisition  Strategy  Panel 
which  consists  of  senior  acquisition  experienced  people.  This  review  is  to  ensure  the  program  plan 
is  thoroughly  vetted.  This  panel  also  recommends  a  source  selection  authority  to  make  the  final 
contract  award  decision.  The  level  of  the  review  members  depends  on  the  size  or  dallar  value  of  the 
program  under  consideration. 

The  Acquisition  Plan  is  considered  a  living  document  and  is  updated  as  required  by  events 
in  the  program.  Often  it  is  either  summarized  or  attached  to  the  Acquisition  Strategy  for  the 
milestone  reviews. 

Source  Selection  Plan  -  The  source  selection  plan  details  the  criteria  to  be  used  to  make  the 
source  selection  decision.  It  identifies  the  specific  criteria  on  which  the  proposals  will  be  judged 
and  the  specific  weights  of  those  criteria.  The  criteria  and  relative  rankings  are  included  in  the 
Request  for  Proposals  for  the  contractors.  Any  factors  not  specifically  contained  in  the  source 
selection  plan  can  not  be  used  to  make  a  source  selection  decision  without  reissuing  the  request  for 
proposals.  This  is  to  ensure  that  the  criteria  are  not  changed  to  favor  a  particular  respondent. 

There  are  two  main  decision  rules  used  in  the  source  selection  process  -  Lowest-priced- 
technically-  acceptable  and  Best-value.  In  the  Lowest-Priced-Technically-Acceptable  decision  the 
agency  has  little  discretion  after  it  identifies  the  technically  acceptable  offers  other  than  to  award  to 
the  lowest  price  competitor.  In  the  Best-Value  case  the  agency  has  significantly  more  discretion  as 
it  compares  and  ranks  the  proposals  considering  capability  and  cost. 

Request  for  Proposals  -  The  Request  for  Proposals  is  the  formal  document  that  tells  the 
contractors  what  the  government  requires.  It  has  two  parts,  the  “Model  Contract”  described  by  the 
Uniform  Contract  Format  contained  in  Parts  I-III  and  the  solicitation  contained  in  Part  IV.  The 
model  contract  follows  the  uniform  contract  format  and  has  several  parts.  Part  I  is  know  as  the 
schedule  and  specifies  the  contract  requirements,  deliverables,  the  schedule,  and  tells  the  contractors 
what  the  government  needs  and  is  intending  to  put  on  contract.  Since  the  contractors  are  expected 
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to  provide  in  their  proposal  a  contract  that  can  be  accepted,  the  offered  contract  from  the  company 
is  expected  to  nearly  match  the  requirements  of  the  government  contract  format  requirements. 

The  second  part  of  the  RFP  is  the  solicitation  and  describes  the  procedures  and  ground  rules 
to  be  followed,  the  basis  for  selection,  and  requests  the  specific  information  needed  to  make  the 
selection.  This  usually  asks  for  a  technical  proposal,  a  management  proposal,  and  a  price  or  cost 
proposal. 


Request  for  Proposal  Package 

The  Model  Contract 
Part  I:  The  Schedule 

Section  A:  Solicitation/Contract  Form 

Section  B:  Supplies  and  Services 

Section  C:  Description/Specification/Work  Statement 

Section  D:  Packaging  and  Marking 

Section  E:  Inspection  and  Acceptance 

Section  F:  Delivery  and  Performance  Section  specifies  time,  place,  and  method  of 
delivery  of  performance  for  items  under  contract. 

Section  G:  Contract  Administrative  Data 
Section  H:  Special  Contract  Requirements 

Part  II:  Contract  Clauses 

Section  I:  General  Contract  Clauses  “Boiler  Plate” 

Part  III:  Lists  Documents,  Exhibits  that  will  be  part  of  contract 

Section  J:  Typically  Work  Breakdown  structure,  Statement  of  Work,  Contract  Data 
Requirements  List,  list  of  specifications,  list  of  Government  Furnished  Equipment 

The  Solicitation 

Part  IV:  Representations  and  Instructions 

Section  K:  Representations,  certifications,  and  other  statements  of  offerors 
Section  L:  Instructions,  conditions  and  Notices  to  Offerors.  Tells  offerors  what  is  to 
be  included  in  their  proposals  and  how  it  should  be  formatted. 

Section  M:  Evaluation  factors  for  award.  Tell  offerors  the  factors  to  be  used  and 
their  relative  importance. 


Table  29:  Format  for  the  Request  For  Proposals  (RFP) 
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The  process  used  for  RFP  development  includes  industry  input.  While  it  has  not  always 
been  the  case,  a  major  emphasis  has  been  placed  on  getting  industry  comments  and  input  on  a  draft 
RFP  prior  to  its  official  release.  This  helps  insure  that  major  issues,  questions,  and  potential  cost 
drivers  are  identified  prior  to  its  release  which  minimizes  the  changes  once  the  formal  RFP  is 
released.  During  this  period  the  contractors  help  identify  the  areas  that  will  be  used  for  the 
evaluation  of  the  proposals.  The  draft  RFP  process  also  allows  the  industry  to  see  the  intentions  of 
the  Air  Force  early  so  they  can  begin  to  develop  their  proposals.  Currently  all  Air  Force  RFP’s  are 
scrubbed  by  the  RFP  support  team  to  ensure  that  they  do  not  tell  the  contractor  how  to  do  things  as 
opposed  to  simply  stating  as  end  objectives  what  needs  to  be  done.  This  centralized  team  at  Wright 
Patterson  Air  Force  Base  is  designed  to  ensure  that  the  acquisiton  reform  initiatives  to  date  are 
included  in  all  new  RFPs. 

The  acquisiton  planning  period  is  finished  with  the  completion  of  the  final  RFP.  The  next 
step  is  the  solicitation  process. 

B.  Solicitation  and  Proposal  Preperation 

The  Federal  Acquisiton  Regulations  call  for  specific  notification  requirements  for  all 
government  contracts.  This  includes  announcing  them  in  the  Commerce  Business  Daily  and 
provides  for  a  specific  period  of  time  before  proposals  are  due.  This  can  be  done  by  publishing  a 
sources  sought  synopsis  which  identifies  potential  companies  early  in  the  process.  Upon  release  of 
the  final  RFP  those  companies  that  have  responded  are  requested  to  bid  and  are  mailed  the  final 
RFP.  Times  to  respond  to  the  final  RFP  are  limited  but  can  be  no  shorter  than  30  days.  Specific 
steps  are  required  to  ensure  that  the  competition  is  open  to  any  qualified  and  interested  company. 
Most  companies  will  be  well  aware  of  the  content  and  schedule  long  before  the  final  release  of  the 
RFP.  The  proposal  is  developed  by  the  contractors  in  response  to  the  RFP.  For  a  description  of  hte 
ocntractors  proposal  process  please  see  appendix.  Once  the  proposals  are  received  the  source 
selection  process  starts. 
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C.  The  Source  Selection  Process 


“The  objective  is  to  select  the  source  whose  proposal  has  the  highest  degree  of  credibility 
and  whose  performance  can  be  expected  to  best  met  the  government’s  requirements  at  an 
affordable  cost.” 

Systems  200  Course  Material 


Upon  receipt  of  the  proposals  the  government  starts  its  source  selection  process.  The  source 
selection  process  is  described  in  the  following  steps.  Prior  to  the  beginning  of  the  source  selection 
process  the  program  office  has  developed  the  acquisition  plan  and  the  source  selection  plan  and  had 
them  approved  by  the  source  selection  authority.  The  source  selection  plan  describes  the  process  to 
be  used  to  evaluate  the  proposals  and  make  the  selection.  It  describes  the  organization,  the 
evaluation  procedures,  and  the  evaluation  criteria.  It  is  approved  by  the  Source  Selection  Authority 
prior  to  the  release  of  the  RFP.  The  selection  criteria  and  their  relative  importance  are  included  in 
the  RFP. 


The  structure  for  the  source  selection  process  includes  a  Source  Selection  Evaluation  Board 
comprised  of  several  panels  to  evaluate  different  aspects  of  each  proposal,  a  Source  Selection 
Advisory  Council,  and  a  Source  Selection  Authority.  The  Source  Selection  Evaluation  Board  is 
solely  responsible  for  evaluating  the  proposals  against  the  set  standards.  The  board  does  not 
evaluate  the  different  proposals  against  each  other.  The  Source  Selection  Advisory  Council  is  made 
up  of  senior  military  and  government  personnel.  They  review  the  SSEB  findings  and  compare  the 
proposals  against  each  other  and  consider  the  contractors’  past  performance.  They  advise  the  source 
selection  authority  of  their  recommendation.  The  final  decision  is  made  by  the  Source  Selection 
Authority.  The  Source  Selection  Authority  is  typically  the  Secretary  of  the  Air  Force  for 
Acquisition  for  ACAT I  and  II  programs  or  the  Acquisition  Center  Commanders  or  the  Program 
Executive  Officers  for  ACAT  III  and  IV  programs. 
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The  Source  Selection  Evaluation  Board  is  divided  into  different  teams.  The  technical  team 
usually  within  the  program  office  typically  forms  the  teams  that  evaluate  the  technical  and 
management  proposals.  The  cost  and  contract  teams  usually  consist  of  cost  analysts  and  the 
contracting  community.  The  Performance  Risk  Assessment  Group  (PRAG)  evaluates  the 
performance  of  the  various  contractors  on  previous  government  contracts  to  provide  a  historical 
perspective  on  the  companies  performance.  Each  team  is  responsible  for  the  evaluation  of  the 
different  proposal  areas.  The  contractor  proposals  are  usually  divided  into  these  same  areas  per 
instructions  in  the  RFP.  The  contract  definitionization  group  includes  the  principle  contracting 
officer  and  is  responsible  to  develop  the  formal  contract  with  each  of  the  companies  that  will  be 
enacted  with  the  company  that  is  selected. 


Majur  Source  Selection  Orgfitizatiop 


Figure  20:  Source  Selection  Organizations  for  Programs  Greater  than  $300  Million  in 
Research  and  Development  Funding 
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Streamlined  Source  Selection  Organization 


Advisors 


Figure  21:  Source  Selection  Organizations  for  Programs  Less  than  $300  Million  Research 
and  Development  Funding. 


Each  team  evaluates  all  the  proposals  against  the  evaluation  criteria.  The  evaluation  criteria 
sets  out  the  exact  aspects  that  each  proposal  will  be  judged  against.  The  three  types  of  criteria 
include  both  specific  criteria  and  cost  or  price  criteria.  The  specific  criteria  are  subdivided  into  areas 
within  the  technical  and  management  areas  such  as  operational  utility  within  the  technical  aspects; 
factors  such  as  airport  operations;  and  subfactors  such  as  runway  requirements.  At  the  lowest  level 
the  specific  criteria  may  include  specific  program  related  characteristics  such  as  minimum  take  off 
distance  at  gross  weight  or  range  at  maximum  payload.  These  are  evaluated  in  a  matrix  fashion  to 
ensure  all  factors  are  rated  and  include  factors  in  the  technical  and  management  sections  of  the 
proposals. 

The  standards  or  assessment  criteria  for  each  factor  and  subfactor  are  set  prior  to  the  receipt 
of  the  proposals.  The  standards  establish  a  baseline  to  measure  each  proposal  to  ensure  they  satisfies 
the  evaluation  criteria.  Each  proposal  is  evaluated  only  against  the  objective  standards  for  each 
factor  and  subfactor.  Specific  criteria  are  established  to  the  minimum  level  that  a  proposal  can  be 
rated  green  or  acceptable.  The  evaluators  are  specifically  prohibited  from  evaluating  the  proposals 
against  each  other. 
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The  evaluation  standards  are  usually  rated  by  color  code  with  the  following  meanings:176 

Blue  -  Exceptional  -  Exceeds  specified  performance  or  capability  in  a  beneficial  way  to  the 
Air  Force  and  has  no  significant  weakness 

Green  -  Acceptable  -  Meets  evaluation  standards  and  any  weaknesses  are  readily  correctable. 

Yellow-  Marginal  -  Fails  to  meet  evaluation  standards;  however,  any  significant  deficiencies 
are  correctable. 

Red  -  Unacceptable  -  Fails  to  met  a  minimum  requirement  of  the  RFP  and  the  deficiency  is 
uncorrectable  without  major  revision  of  the  proposal. 

The  color  coding  of  the  evaluation  criteria  is  weighted  to  higher  levels.  Any  red  criteria 
would  carry  to  the  highest  level  for  that  area.  During  the  technical  evaluation  deficiency  reports  are 
developed  for  each  item  in  each  proposal  that  does  not  meet  the  specific  evaluation  criteria.  These 
reports  are  included  in  the  final  briefing. 

The  cost  factors  are  evaluated  for  realism,  completeness,  and  reasonableness.  The  bottom 
line  price  will  be  determined  as  well  as  cost  analysis  of  each  element  of  the  proposal.  Also 
accomplished  is  cost  and  fee  evaluation  and  profit  fee  analysis  to  ensure  that  the  weighted 
guidelines  are  not  exceeded. 

These  findings  are  then  reviewed  by  the  Source  Selection  Advisory  Council  who  review  and 
comment  on  them.  The  council  then\  compares  the  various  proposals  against  each  other,  looks  at  the 
past  performance  of  each  company,  and  evaluates  the  risks  associated  with  each  proposal  and 
company. 

The  risk  based  on  past  of  performance  of  the  company  and  the  risk  involved  in  achieving  the 
objectives  in  the  specific  proposal  is  assessed  with  verbal  expressions  of  High,  Moderate,  and  Low. 
Risk  here  is  a  level  of  confidence  that  the  job  can  be  performed  as  proposed.  Performance  risk  is 
based  on  the  company’s  performance  on  similar  previous  contracts.  It  is  accomplished  by  the  PRAG 
and  indicates  the  Air  Force  assessment  of  the  likelihood  of  the  company  to  achieve  the  proposed 
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outcomes  on  a  company  basis.  The  proposal  risk  is  the  risk  associated  with  aspects  associated  with 
the  specific  proposal.  High  risk  indicates  that  significant  doubt  exists,  moderate  indicates  that  some 
doubt  exists  and  low  means  that  little  doubt  exists.  These  ratings  accompany  the  color  ratings. 

Briefings  are  developed  demonstrating  the  evaluation,  the  strengths,  weaknesses  and  cost  of 
each  proposal  and  are  presented  to  the  Source  Selection  Authority.  At  this  point  a  selection  can  be 
made  directly  and  the  contract  can  be  awarded  without  discussion  to  either  the  contractor  with  the 
lowest  price  or  the  one  with  no  deficiencies.  Typically,  however,  there  is  a  determination  of  the 
competitive  range  followed  by  further  discussions  with  those  selected.  Determination  of 
competitive  range  is  based  on  the  evaluation  factors  and  includes  offerors  with  a  chance  to  win.  It 
is  a  subjective  decision  by  the  Source  Selection  Authority  and  the  contracting  officer.  Several 
contractors  are  eliminated  that  are  not  likely  to  be  selected  because  of  cost  or  technical  reasons  in 
their  proposal.  Following  this  determination  the  Contracting  Officer  can  release  Discrepancy 
Reports  (DRs)  and  Clarification  Requests  (CRs)  to  the  remaining  contractors  and  request  updated 
and  additional  information  to  address  issues  not  adequately  covered  in  the  original  proposal.  Care 
is  taken  as  to  ensure  that  information  is  not  spread  between  contractors  and  that  “  Technical 
Leveling”  (coaching  from  the  government)  or  technical  transfusion  (sharing  of  proprietary 
information)  between  proposals  does  not  occur.  It  is  also  not  intended  to  be  an  auction  by  revealing 
competitor’s  price  position,  or  win  price.  This  period  of  discussion  is  intended  to  ensure  that  the 
proposals  accurately  reflect  the  Air  Force  desires  and  needs.  After  the  discussions,  a  call  for  the 
Best  And  Final  Offers  (B  AFO)  is  given  and  the  companies  submit  their  final  proposals  and  pricing 
information.  These  updated  proposals  are  then  re-evaluated  and  final  evaluations  are  briefed  and 
submitted  to  the  source  selection  authority  for  the  final  decision. 
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Example  of  Matrix  Evaluation 

Contractor  AAA  AFFARS  Appendix  AA 

Ant:  Technical 
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Arrows  indicate  movement  between  initial  rating  and  BAFO  rating  mm 

Figure  22:  Example  of  Matrix  Evaluation  Presented  to  a  Source  Selection  Authority177 
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The  source  selection  authority  makes  the  selection  based  on  one  of  two  decision  rules,  best 
value  or  lowest  cost  technically  acceptable  proposal.  The  Source  Selection  Authority  is  bound  by 
the  stated  evaluation  factors  and  the  stated  relative  levels  of  importance.  He  is  also  constrained  by 
many  regulations  and  statues.  In  making  the  source  selection  decision  the  SSA  may  ignore  the 
scores,  rescore  the  proposals,  or  declare  differences  insignificant.  The  selection  is,  however, 
susceptible  to  review  under  protests.  Once  the  selection  has  been  made  the  Source  Selection 
Authority  notifies  the  appropriate  government  and  congressional  offices  and  then  the  award 
announcement  is  made. 

Following  the  award  announcement  the  losing  bidders  are  debriefed  as  to  their  ratings  and 
evaluation  only  and  the  general  reasons  they  were  not  selected.  They  are  not  briefed  on  other 
competing  contractors’  proposals.  Losing  contractors  can  protest  the  decision  to  the  General 
Accounting  Office,  Court  of  Federal  Appeals,  or  District  Court.  While  protests  occur  frequently, 
few  are  were  upheld.  Of  the  47  protests  in  1995  only  2  were  upheld.  The  Air  Force  goes  to  great 
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lengths  to  ensure  that  the  source  selection  process  is  “fire  proof’  and  can  withstand  a  protest  often 
at  great  length  and  expense. ,7* 


Provided  there  are  no  protests  of  the  award,  the  contract  can  then  be  signed  and  the 
contractor  can  begin  the  development  effort.  The  contractor  will  not  be  reimbursed  for  efforts  on 
the  specific  product  that  it  may  have  spent  up  to  this  point.  It  will  be  reimbursed  for  bid  and 
proposal  money  through  its  overhead  rates  on  existing  contracts.  Industry  Research  and 
Development  Funds  (IR&D)  are  not  allowed  to  be  reimbursed  if  they  are  used  on  a  specific  product. 
They  can  be  used  for  generic  efforts  that  build  the  technology  to  be  used. 


178  One  example  would  be  a  best  value  source  selection  that  was  accomplished  at  ESC.  A  winning  contractor  was 
selected  but  not  announced.  A  major  consulting  company  was  hired  to  ensure  impartiallity  and  review  the  best  value 
selection  process.  After  a  year  the  same  contractor  was  selected  and  the  contract  was  awarded. 
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A.  Program  Office  Survey 

B.  Contractor  Survey 

C.  Pentagon  Survey 


481 


Survey  of  Defense 
Product  Development  Practices 

©  Massachusetts  Institute  of  Technology,  1997 
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Massachusetts  Institute  of  Technology 
Lean  Aircraft  Initiative 


Report  Control  Symbol  RCS:  SAF-AQX(OT)9701 
US  Air  Force  Survey  Control  Number:  (USAF  SCN)  96-98 


Suggested  Products  and  Projects  by  Organization 


Organization  Project  Name 

SOFPARS 

ASC/LAAV  C-32  (VC-X) 

ASC/LBA  C-130J  Aircraft 

ASC/LBA  C-l  30  H2  Aircrew  Training  System 

ASC/LCA  C/KC-135  Improved  Aerial  Refueling  System  (IARS) 

Multipoint  Refueling  Sytem  (PPRS) 

ASC/LDAA  A  Broad  Based  Environment  for  Test  (Abbet) 

ASC/LN  Advanced  Strategic  and  Tactical  Infrared  Expendables 

ASC/LN  AN/ALE-47  Countermeasure  Dispensing  Systems 

ASC/LNT  Joint  Service  Electronic  Combat  Systems  Tester  (JSECST) 

ASC/LNW  Common  Missile  Warning 

ASC/LNY  Large  Aircraft  Directed  Infrared  Countermeasures 

(DIRCM) 

ASC/LPC  FI  17  Engine  -  Propulsion  for  C-17  Aircraft 

ASC/LPJ  ENGINE  MODEL  DERIVATIVE  PROGRAM 

ASC/LPP  Engine  Component  Improvement 

ASC/LU  Combat  Talon  II  (MC- 1 30H) 

ASC/LU  CV-22 

ASC/LU  DIRECED  INFRARED  COUNTERMEASURES 

(DIRCM) 

ASC/LU  Quiet  Knight  (SOF) 

ASC/LUR  INTEGRATED  AVIONICS  TEST  (RFMETS) 

ASC/LUU  Gunship  (AC-130U) 

ASC/LY  AVIONICS  STRATEGIC  PLANNING  IPT 

ASC/LY  JOINT  HELMET  MOUN  TED  CUEING  SYSTEM 

ASC/LYB  Embedded  GPS/INS 

ASC/QLAX  F-117A  Aircraft 

ASC/RA  Big  Safari 

ASC/RA  SR -71  Reactivation 

ASC/RAJ  SR-71  Joint  Airborne  Sigint  System 

ASC/RAPAU  HAE  UAV 

ASC/RFMC  Propulsion  Development  Systems  Office  (DSO) 

ASC/SM  Coatings  Technology  Integration 

A  SC/VC  AGM-129/Advanced  Cruise  Missile  (ACM) 

ASC/VJ  Tri-Service  Standoff  Attack  Missile  (TSSAM) 

ASC/VX  DSU-33 

ASC/VX  Joint  Programmable  Fuze  (JPF) 

ASC/VXC  Aerial  Targets 

ASC/XR  Joint  Advanced  Strike  Technology  Program 

ASC/YD  B-1B  Aircraft  Spo  Director 

ASC/YD  B-1B  Aircraft  Conventional  Munitions  Upgrade  Program 

-  Cluster  Bomb  Integration 
ASC/YDQC  B-1B  Computer  System  Upgrade 

ASC/YDQD  B-1B  Defensive  System  Upgrade  Program 

ASC/YFF  F-22  Aircraft  Air  Vehicle 

ASC/YFFU  F-22  Aircraft  F-l  19  Engine 

ASC/YFFW  F-22  Aircraft  Program  Director 

ASC/YG  AMRAAM  AIM-I20A/B/C/  MISSILE 

ASC/YH  Joint  Stand-off  Weapon  (JSOW)  (BLU-108/B) 

ASC/YH  Wind  Corrected  Munitions  Dispenser 

ASC/YN  Non-Developmental  Airlift  Aircraft  (NDAA) 

ASC/YPR  F-16  Night  Vision  Imaging  System  (NVIS) 

ASC/YPR  F-16  Common  Configuration  Implementation  Program 

ASC/YPR3  F-16 

ASC/YPR3  Targeting  Pod  for  F-16  Block  30 

ASC/YPR4  Common  Missile  Warning  Svstem/40T6  Program 

ASC/YPR6  F-16RWR 

ASC/YPX  F-16FMS 

ASC/YS  B-2  Aircraft  Integration 

ASC/YSB  B-2  Aircraft  Production  Program 

ASC/YSBZ  B-2  Engine 


Organization  Project  Name 

ASC/YSTC  B-2  Aircraft  Training  System 

ASC/YTJ  Joint  Primary  Aircraft  Training  System  (JPATS)  T-1A 

Training  System 

ASC/YU  Joint  Direct  Attack  Munition 

ASC/YWMA  C-1 41  Aircrew  Traiping  System 

ASC/YWMC  JSTARS  Flight  Crew  Training  System 

ASC/YWME  KC-135  Aircrew  Training  System 

ASC/YWMG  C-17  Air  Crew  Training  System 

ASC/YWMT  T-2S  Simulator  For  Electronic  Combat  Training  (T-2S 
SECT) 

ASC/'’WP  Air  Combat  Command  Training  Program 

ASC/YWPA  T-3A  Enhanced  Flight  Screener  (EFS)  Aircraft 

ASC/YWPC  B-1B  Training  System 

ASC/YWPD  T-3A  Enhanced  Flight  Screener  (EFS)  Aircraft 

ESC/AV  AMC  C2  IPS  Information  Processing  Systems 

ESC/AV  Command  and  Control  SPO 

ESC/AV  Ground  Theater  Air  Control  System 

ESC/AW  AWACS  Program  Office  SPO 

ESC/A WD  Theatre  Missile  Defense 

ESC/A WIS  Saudi  AWACS 

ESC/AWJ  767  AWACS 

ESC/AWN  RSIP 

ESC/AWZ  Communications  Extend  Sentry 

ESC/AWZ  Engine  Extent  Sentry 

ESC/A WZP  PDM  Re-Engineering 

ESC/IC  Base  Information  Protect 

ESC/IC  Combat  Intelligence  System  (CIS) 

ESC/IC  IDHS  -  Automated  Message  Handling  System 

ESC/IC  IDHS  -  IC4I 

ESC/IC  IDHS  -  MAXI 

ESC/IC  IDHS  -  Sentinel  II  Integration 

ESC/IC  Intelligence  and  Information  Warfare  SPO 

ESC/IC  Intelligence  Receivers  -  MATT 

ESC/IC  R/IGS  -  Eagle  Vision/Commercial  Systems 

ESC/IC  Voice  Processing  Training  Systems  Replacement 

ESC/ICD  Intelligence  Data  Handling  Systems  Product  Group 

ESC/JS1C  Joint  Stars 

ESC/JS1M  Joint  Stars  -  Deployable  Mission  Planning 

ESC/JSS1D  Joint  Stars  -  TADIL-J 

ESC/MCB  MILSTAR  Terminal  Command  Post  Production 

ESC/SRM  Alternate  Processing  Correlation  Center 

ESC/5RM  Command  Center  Processing  and  Display  Systems 

ESC/TD  JTIDS 

ESC/TG  DoD  Advanced  Automation  Systems 

ESC/TG  HIDAR  &  MEECN  Termination/  Sustainment 

ESC/TG  Modified  Miniature  Receive  Terminal 

ESC/TG  Pacer  Speak 

ESC/TG  Theater  Deployable  Communications  (TDC) 

ESC/TG  Voice  Communications  Switch  Systems 

ESC/TN  Advanced  Interface  Control  Unit  (AICU) 

ESC/TN  Space  Surveillance  Network  Improvement 

ESC/IR  Surveillance  and  Control  SPO 

ESC/YV  B-1B  Mission  Planning 

ESC/YV  B-2  MSN  Planning 

ESC/YV  Common  Mapping  Production  System 

ESC/YV  Mission  Data  Preparation  System 

ESC/YV  Mission  Planning  Systems  (MPS) 

ESC/YV  Peace  Fenghuang,  AFMSS 

ESC/YV  Peace  Fox  VI,  AFMSS 

ESC/YV  Peace  Sun  IX,  AFMSS 

SA/ALC  FI  17-PW-100  Engine  -  Propulsion  for  C-17  Aircraft 


Instructions  * 


This  survey  is  part  of  a  large,  on-going  research  program  involving  the  US  Air  Force,  a 
number  of  firms  in  the  defense  aerospace  sector,  and  academia.  The  research  program  is 
investigating  the  application  of  “lean”  practices  to  the  defense  industry.  This  survey  is  designed 
specifically  to  assess  the  impact  of  various  attributes  of  the  Department  of  Defense  (DoD) 
acquisition  environment  on  product  development  practices  at  the  project/product  level. 

Your  answers  and  cooperation  are  vital  to  the  success  of  this  study!  Information  on  the 
issues  addressed  by  this  study  is  not  available  from  any  other  source,  so  your 
assistance  is  vital  to  gaining  an  understanding  of  the  issues  we  are  trying  to 
address. 

Please  select  a  single  significant  development  effort  (recently  planned,  on¬ 
going,  or  recently  completed)  contained  in  the  programs  that  you  are  responsible 
for  and  answer  the  questions  as  they  refer  to  that  development  effort.  (If  possible 
select  from  the  suggested  list  on  the  opposite  page.)  For  example,  Capt  Geurts  is  an  action  officer 
associated  with  the  AW  ACS  program.  He  would  select  one  of  the  development  efforts  contained  in 
the  AWACS  program  such  as  the  Radar  System  Improvement  Program  (RSIP),  with  which  he 
may  be  the  most  familiar.  He  would  then  answer  the  questions  as  they  apply  to  the  RSIP  program 
only.  A  development  effort  may  include  the  entire  program  such  as  the  F-22  or  may  be  a  smaller 
integration  or  upgrade  effort  on  established  programs.  If  the  selected  project  is  part  of  a  larger 
program  office,  do  not  answer  the  questions  as  they  might  apply  to  that  larger  organization,  but 
only  as  they  apply  to  that  particular  project  or  product  development  effort.  If  you  have  any 
questions  on  this  matter  please  do  not  hesitate  to  call  or  e-mail  me  at  the  address  below. 

Please  be  objective  and  precise  in  your  responses.  We  understand  that  you  may  have 
concerns  about  confidentiality.  Several  measures  have  been  taken  to  ensure  that  your  responses 
will  remain  confidential.  Nobody  but  the  researchers  associated  with  the  Lean  Aircraft  Initiative 
will  have  access  to  the  information  requested  in  this  survey.  Names  and  program  data  will  be 
separated  from  response  data.  All  analysis  of  the  survey  data  will  be  presented  in  the  form  of 
aggregated  statistics.  No  individuals  or  individual  programs  will  be  identified  in  the 
analysis  or  reporting  of  the  results.  We  understand  that  the  success  of  any  research 
depends  upon  the  quality  of  the  information  on  which  it  is  based,  and  we  take  seriously  our 
responsibility  to  ensure  that  any  information  you  entrust  to  us  will  be  protected  with  the  same  care 
that  you  would  use  yourself. 

This  survey  has  been  designed  in  four  sections.  Section  A  asks  questions  relating  to 
program  planning  and  execution.  Section  B  asks  more  detailed  questions  about  the  cost  and  cost 
estimate  development  for  this  program.  Section  C  asks  about  schedule  and  schedule  related 
planning  aspects  of  this  project.  Section  D  asks  for  some  general  information  about  you  and  your 
organization.  The  survey  has  been  pretested  and  should  take  no  longer  than  one  hour  to  an  hour 
and  a  half  to  complete. 

When  you  have  completed  the  survey,  please  enclose  it  in  the  accompanying  envelope  and 
return  it  directly  to  MIT.  If  you  would  like  a  copy  of  the  results  of  the  survey,  place  a  check  mark 
in  the  space  provided  at  the  end  of  the  survey. 

Thank  you  for  your  help.  If  you  have  any  questions  about  this  survey  or  the  objectives  of 
this  research,  feel  free  to  contact  me  at  any  time: 


Capt.  Ross  McNutt 
Lean  Aircraft  Initiative 
Massachusetts  Institute  of  Technology 
18  Vassar  Street,  Room  20A-0 11 
Cambridge,  MA  02139 


Office:  (617)  258-6594 
Home:  (617)  641-0590 
Fax:  (617)  253-4235' 
E-mail:  rncnuttrt@mit.edu 


♦Please  answer  all  questions  as  they  relate  to  a  specific  development  project  or  product 

Section  A:  Program/Product  Information  and  Objectives 


A .  1  Please  indicate  your  organization  and  role  in  this  product/project? 

Org:  □  HQ/XO  □  SAF/AQ  □  Other:. 

Role:  □  Program  Element  Monitor  □  Action  Officer  □  Other:. 

A.2  Project/Product  Information 

Project/Product  Name:  _ 

Overall  Program  Name:  _ _ 

Involved  Development/Logistics  Center: _ 


Example: 

(RSIP) 

(AWACS) 

(ESC) 


A. 3  What  is  the  DoD  acquisition  category  (ACAT)  designation  for  this  project  and  the  overall 

program:  (ID,  IC,  II,  II,  IV/V) 

Example: 

Specific  Product  _  (RSIP-  IC) 

Overall  Program  _ _  (AWACS  -  ID) 


A .  4  In  what  acquisition  phase  is  this  project/product  currently?  (If  not  designated  an  official 
phase  please  estimate.) 

□  Pre-Milestone  0  □  Engineering  &  Manufacturing  Development 

□  Concept  Exploration  &  Definition  □  Production  &  Deployment 

□  Demonstration  &  Validation  □  Other: _ 

A .  5  Which  of  the  following  categories  best  describes  the  type  of  system  this  project  is  developing? 

□  Aircraft  (airframe  and  mechanical  systems)  □  Electronic  system  (non-aircraft) 

□  Aircraft  (avionics  and  electronic  systems)  □  Missile  or  munitions 

□  Aircraft  (propulsion)  □  Software-dominated  system 

□  Spacecraft  or  launch  system  □  Other: _ 

A.  6  In  your  judgment,  when  was  this  particular  project  started? _ (month/year) 

What  milestone  or  event  signified  the  starting  point: _ 

A .  7  When  was  this  particular  project  approved  as  an  acquisition  project  (Milestone  1  or  equivalent)? 

_ (month/year) 

A .  8  When  is  the  first  production  item  or  its  equivalent  planned  to  be  completed? 

_ (month/year) 

A. 9  Approximately  how  long  is  this  project  expected  to  operate  in  its  delivered  configuration, 
not  including  major  modifications  or  changes? 

_ years  or  until  the  year _ (ex.  2015) 


A.  10  Please  estimate  the  extent  of  the  technological  advance  required  in  the  product  technology 
components)  that  are  considered  critical  to  fulfilling  this  project’s  technical  objectives: 


All  Non- 
Developmental 
Items 

1  2 


Some 

Development 

Required 

3  4  5 


All  New 
Development 
Required 

6  7 


A.  11  To  what  extent  do  commercial  markets  currently  exist  for  the  key  or  enabling  product 
technologies  being  used  in  this  project? 

No  Some  Many 

Markets  Markets  Markets 

1  2  3  4  5  6  7 

A.  12  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
product  technologies  required  by  this  project,  relative  to  existing  systems: 

□  Revolutionary  new  core  technologies  or  concept 

□  New  generation  of  product  architecture  or  platform. 

□  Incremental  improvements  to  an  existing  generation  of  product  architecture  or  platform. 

□  Little  or  no  change  to  the  existing  product  technology. 


A.  13  Please  rank  the  order  of  importance  of  the  following  considerations  in  meeting  this 
project’s  objectives  (“1”  indicates  the  item  with  the  greatest  importance  in  meeting  this 
project’s  objectives,  “4”  indicates  the  item  with  the  least  importance  in  meeting  this 
project’s  objectives): 

_  Low  acquisition  cost. 

_  Low  operational  and  support  cost. 

_  Short  schedule  to  reach  operational  capability. 

_  Superior  technical  performance. 


A.  14  To  what  extent  have  the  following  items  associated  with  this  project  changed  since 
Milestone  I  (or  project  start)? 

No  Change  Some  Change  Entirely  New 


Performance  Requirements 
Mission  Concept 
Threat/Adversary 


12  3  4 
12  3  4 
12  3  4 


5  6  7 
5  6  7 
5  6  7 


Budget  Profile 
Program  Schedule 
Program  Priority 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


6  7 

6  7 

6  7 


A .  1 5  In  response  to  future  unforeseen  events,  to  what  extent  are  you  likely  to  change  each  of  the 
following  in  order  to  achieve  the  overall  program  objectives? 

Least  Likely  Somewhat  Likely  Most  Likely 
to  Change  to  Change  to  Change 


Total  project  cost 

Total  project  schedule  length 

Product  system  performance 


1  2 
1  2 
1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


A.  16  From  each  of  the  following  pairs  of  statements,  please  select  the  one  from  each  row  which 
best  represents  this  project: 


_ Statement  A _ or _ Statement  B  _ 

□  This  project  was  started  in  response  to  an  □  This  project  was  started  in  response  to  a 
identified  deficiency  or  need  from  an  new  technology  development  that  allowed 

operational  user.  a  new  capability  that  the  user  desired. 


□  This  project  was  started  as  the  result  of  the  □  This  project  was  started  at  a  senior  leader’s 
normal  service  planning  process.  direction. 


□  This  project  followed  the  typical  □  This  project  did  not  follow  the  typical 
development  cycle  as  described  by  the  development  cycle  as  described  by  the  DoD 

DoD  5000  series.  5000  series. 


□  This  project  is  intended  to  meet  a  current  □  This  project  in  intended  to  meet  a  future 
operational  deficiency  or  projected  operational  deficiency. 


□  This  project  has  a  specific  calendar  date  □  This  project  does  not  have  a  specific 
when  it  is  needed.  calendar  date  when  it  is  needed. 


□  This  project  used  a  prototype  phase  prior  to  □  This  project  did  not  use  a  specific  prototype 
start  of  EMD  to  demonstrate  a  capability.  phase  and  proceeded  directly  with  EMD. 


□  This  project  is  an  upgrade  to  an  existing  □  This  project  represents  a  new  system, 

system. 


□  This  project  has  the  significant  personal  □  This  project  does  not  have  the  significant 

interest  of  a  high  level  “sponsor”.  personal  interest  a  high  level  “sponsor”. 


□  This  project  is  run  by  an  average  System  □  This  project  is  run  by  a  particularly  influential 
Program  Director  or  equivalent.  or  effective  System  Program  Director. 


□  This  project  is  run  by  an  average  program  □  This  project  is  run  by  a  particularly  influential 
manager.  or  effective  program  manager. 


□  The  schedule  of  this  project  is  funding 
limited. 


□  The  schedule  of  this  program  is 
technology  and  engineering  limited. 


□  This  project  has  a  significant  sense  of  0  This  project  does  not  have  a  significant 
urgency  about  it.  sense  of  urgency  about  it. 


□  This  project  currently  has  a  well  defined  □  At  this  time,  there  are  no  specific  or 

P3I  or  technology  insertion  strategy.  definitized  plans  to  upgrade  this  project. 


□  There  are  other  competing  systems  or 
projects  that  can  meet  this  project’s 
objectives. 


□  There  are  no  other  competing  systems  or 
projects  that  can  meet  this  project’s 
objectives. 


A.  17  Does  the  funding  and/or  management  of  this  project/product  involve  another  service, 
government  agency,  or  country? 

□  Other  Service  □  Other  Government  Agency  □  Other  Country 


A .  1 8  Please  estimate  the  priority  placed  on  this  project  in  each  of  the  following  organizations: 


Lowest 

Priority 


Average  Highest 

Eriurity  Priority 


The  User 

The  Program  Office 
The  Service 
TheDoD 
The  Congress 


1  2  3 
1  2  3 
1  2  3 
1  2  3 
1  2  3 


4  5  6  7 
4  5  6  7 
4  5  6  7 
4  5  6  7 
4  5  6  7 


A .  1 9  In  your  opinion,  please  rate  the  “Sponsor”,  the  Program  Director,  the  Government  Program 
Manager,  and  the  Contractor  Program  Manager  for  this  project  with  respect  to  their  effectiveness 
and  influence  at  getting  things  accomplished  within  the  DoD,  the  service,  and  the  contractor 
organizations  with  respect  to  their  peers? 


Program  “Sponsor”  or  advocate 

N/A  Weak 

0  1 

2 

Axgiagg 

3 

4 

Sir one 

5 

Truly 

Exceptional 
6  7 

Program  Director  or  equivalent 

0 

1 

2 

3 

4 

5 

6  7 

Government  Project  Manager 

0 

1 

2 

3 

4 

5 

6  7 

Contractor  Program  Manager 

0 

1 

2 

3 

4 

5 

6  7 

A. 20  Have  the  performance  capabilities  of  any  of  the  underlying  technologies  that  you  consider 
to  be  key  to  the  effectiveness  of  your  system  (e.g.,  digital  computer  processors,  advanced 
materials)  changed  significantly  during  the  duration  of  your  program,  exclusive  of  any 
technological  advances  made  by  this  program? 

No  Change  Some  Changes  Major  Chants 

1  2  3  4  5  6  7 


A .  2 1  Is  this  project/product  considered  “fast  track”  (the  schedule  has  officially  been  accelerated 
so  as  to  shorten  the  total  acquisition  cycle)? 

□  No  □  Yes 


A. 2 2  To  what  extent  is  a  short  acquisition  cycle  (quick  initial  operational  capability)  an  important 
part  of  this  project’s  overall  objectives? 

Not  Somewhat  Very 

Important  Important  Important 

1  2  3  4  5  6  7 


A. 2 3  Please  indicate  the  extent  to  which  you  believe  the  Program  Office  has  incentives  to  try 
to  achieve  the  following  objectives  for  this  project: 


Meet  cost,  schedule,  performance  goals 


A  Strong 

Disincentive 
1  2 


No 

Incentive 

3  4  5 


A  Strong 

Incentive 
6  7 


Reduce  total  program  cost  below  target  l  2 

Reduce  unit  cost  below  target  1  2 

Complete  program  ahead  of  schedule  1  2 

Exceed  planned  performance  requirements  1  2 

Exceed  reliability  and  maintenance  requirements  1  2 


Please  briefly  indicate  the  nature  or  form  of  these  incentives: 


3 

3 

3 

3 

3 


4  5  6  7 
4  5  6  7 
4  5  6  7 
4  5  6  7 
4  5  6  7 


A. 24  Please  indicate  the  extent  to  which  you  believe  the  contractor  has  incentives  to  try  to 


achieve  the  following  objectives  for  this  project: 

A  Strong 
Disincentive 

No 

Incentive 

A  Strong 

Inc&ntm 

Meet  cost,  schedule,  performance  goals 

l  2 

3 

4 

5 

6  7 

Reduce  total  program  cost  below  target 

l  2 

3 

4 

5 

6  7 

Reduce  unit  cost  below  target 

1  2 

3 

4 

5 

6  7 

Complete  program  ahead  of  schedule 

1  2 

3 

4 

5 

6  7 

Exceed  planned  performance  requirements 

1  2 

3 

4 

5 

6  7 

Exceed  reliability  and  maintenance  requirements  1  2 

Please  briefly  indicate  the  nature  or  form  of  these  incentives: 

3 

4 

5 

6  7 

A. 2 5  Please  indicate  the  extent  to  which  you  believe  the  user  has  incentives  to  try  to  achieve  the 


following  objectives  on  this  project: 

A  Strong 

No 

A  Strong 

Disincentive 

Incentive 

Incentive 

Meet  cost,  schedule,  performance  goals 

1  2 

3 

4 

5 

6  7 

Reduce  total  program  cost  below  target 

1  2 

3 

4 

5 

6  7 

Reduce  unit  cost  below  target 

1  2 

3 

4 

5 

6  7 

Complete  program  ahead  of  schedule 

1  2 

3 

4 

5 

6  7 

Exceed  planned  performance  requirements 

1  2 

3 

4 

5 

6  7 

Exceed  reliability  and  maintenance  requirements 

1  2 

3 

4 

5 

6  7 

Please  briefly  indicate  the  nature  or  form  of  these  incentives 


Section  B:  Program  Cost  and  Cost  Planning  Information 


B .  1  What  is  this  project’s  current  estimate  or  actual  budget  expenditure  and  what  were  the  initial 
planned  expenditures  for  all  program  expenses  in  each  of  the  following  program  phases? 
Use  actual  or  projected  then  year  dollars. 

CURRENT  ESTIMATE  INITIAL  PLAN 


Pre-EMD 

EMD 

Production 


($  million) 
($  million) 
($  million) 


($  million) 
($  million) 
($  million) 


B.2  What  were  the  actual  and  initially  planned  development  (Pre-EMD  and  EMD)  funding 
levels  for  this  project  from  all  sources  for  the  following  years:  (Then  Year  $) 


FY  <88  88  89  90  91  92  93  94  95  96 

Actual  levels  _  _  _  _  _ 

Initial  Project  Plan  _  _  _  _  _  _ 


($M) 

($M) 


B .  3  What  are  the  projected  future  development  funding  levels  for  this  project:  (Then  Year  $) 

FY  97  98  99  00  01  02  03  04  05  >05 


Current  Plan 
Initial  Project  Plan 


($M) 

($M) 


B .  4  Please  estimate  the  expected  change  to  date  in  this  project’s  development  cost  (Pre-EMD 
and  EMD  from  all  sources)  using  the  initial  program  plan  as  a  baseline.  Calculate  this 
as  the  amount  and  percentage  change  in  the  current  projected  cost  compared  with  the  initial 
projected  cost  for  the  entire  development  program.  Treat  increases  beyond  original 
plan  as  a  positive  change  in  cost  Use  then  year  dollars. 

Total  development  cost  change  _  (+/-  $  million  to  date) 

Total  development  cost  change  percent  _  (+/-%  to-date) 

Now,  please  estimate  the  percentage  of  this  project’s  cost  change  (calculated  above)  that 
can  be  attributable  to  unplanned  changes  in  your  requested  annual  funding  levels  (funding 
instability),  changes  in  the  program’s  technical  requirements,  technical  problems,  or  other 
reasons,  respectively.  The  sum  of  the  proportions  should  equal  100%. 

%  of  cost  change  due  to  funding  instability  _ (+/-  %) 

%  of  cost  change  due  to  requirements  changes _ _(+/-%) 

%  of  cost  change  due  to  technical  problems  _ (+/-%) 

%  of  cost  change  due  to  other  reasons  _ (+/-%)(please  specify:) _ 


100% 


B .  5  What  are  the  current  projected  and  what  were  the  initial  projected  unit  cost  and  the  number 
of  items  to  be  produced  for  this  project.  Please  estimate  the  unit  or  “fly-away”  price  in 
then  year  $. 

Current  Projected  Unit  Cost  _  Initial  Projected  Unit  Cost _ 

Current  Planned  Quantity  _  Initial  Planned  Quantity  _ 

B .  6  What  type  of  contract  does/did  your  program  have  with  your  contractor  in  the  development 
phase  for  this  project? 

□  Fixed  Price  □  Cost  Plus  0  Other _ 

B .  7  Was  the  project/product  development  contract  awarded  on  the  basis  of  a  sole  source  or  a 
competitive  bidding  process? 

□  Sole  Source  □  Competitive  Bid  (Number  of  competitors: _ ) 

B  .8  Was  the  development  contract  for  this  project/product  part  of  a  contract  modification  of  a 
larger  contract? 

□  No  □  Yes 

If  yes,  please  state  the  reason  for  the  larger  contract: _ _ _ 

B.9  By  what  percentage  did/does  this  project’s  annual  budget  allocation  differ  from  that 
requested  in  the  project’s  plan,  if  any?  Use  +  /  -  to  indicate  increases  or  decreases  from 
plan,  respectively. 

This  year _ (%)  On  average  _ (%) 

B.10  To  what  extent  would  you  say  that  your  program’s  progress  in  meeting  its  overall 
objectives  during  this  program  phase  has  been  affected  by  program  instability  (defined  as 
unplanned  or  unanticipated  changes  in  your  requested  annual  budget  or  other  critical 
resources,  program  schedule,  or  definition  of  work)? 

No  Effect  Some  Effect  Major  Effect 

1  2  3  4  5  6  7 

B .  1 1  How  many  times  was  this  project  re-baselined,  re-phased,  or  significantly  altered  since  it 
began? 

Total  number  of  program  baselines  or  project  plans:  _ 

.  Please  estimate  how  many  of  the  new  baselines  or  plans  above  changed  your  programs 
scope/requirements,  deliverables,  funding  (include  all  types  of  funding  changes),  or 
schedule?  List  each  type  of  change  by  the  program  phase  in  which  it  occurred  in  the  matrix 
below.  Please  estimate  if  you  don’t  know  the  exact  number  of  changes  of  each  type. 


Program 

Phase 

Number 
of  Plans 

Funding 

Pre-EMD 

EMD 

Production 

Comments: 


B.12  Please  indicate  the  extent  to  which  you  believe  the  following  items  impacted  the  initial 
funding  profile  as  it  was  developed  at  the  Pentagon  level  for  this  project : 


No 

Impact 

Historical  similar-program  data  1  2 

3 

Some 

Impact 

4 

5 

6 

Defining 

Impact 

7 

Parametric  modeling  or  estimation 

1  2 

3 

4 

5 

6 

7 

Expert  judgment 

1  2 

3 

4 

5 

6 

7 

Bottoms  up  cost  analysis 

1  2 

3 

4 

5 

6 

7 

Comparable  commercial  development 

1  2 

3 

4 

5 

6 

7 

User  proposed  funding  profile 

1  2 

3 

4 

5 

6 

7 

SPO  proposed  funding  profile 

1  2 

3 

4 

5 

6 

7 

Contractor  proposed  funding  profile 

1  2 

3 

4 

5 

6 

7 

Service  programming  and  budgeting  process 

1  2 

3 

4 

5 

6 

7 

Expected  development  funding  availability 

1  2 

3 

4 

5 

6 

7 

Expected  production  funding  availability 

1  2 

3 

4 

5 

6 

7 

Expected  service  budget  levels 

1  2 

3 

4 

5 

6 

7 

Other:  .  _ 

1  2 

3 

4 

5 

6 

7 

B.13  In  your  opinion,  what  was  the  level  of  influence  of  the  following  organizations 
developing  the  government  initial  funding  profile  for  this  project? 


Not 

Some 

Sole 

Involved 

Influence 

Influence 

in 


Users 

Program  Office 

FFRDC  (Mitre,  Aerospace,  etc.) 
PEO 

Service  Leadership 
Service  Acquisition  Community 
Service  Cost  Analysis  Group 
Service  Planning  Community 
Service  Budget  Community 

DoD  Leadership 
DoD  Acquisition  Community 
DoD  Budget  Community 
DoD  Planning  Community 

Potential  Contractors) 

Potential  Subcontractors 
Other: _ 


12  3  4 
12  3  4 
12  3  4 
12  3  4 

12  3  4 
12  3  4 
12  3  4 
12  3  4 
12  3  4 

12  3  4 
12  3  4 
12  3  4 
12  3  4 

12  3  4 
12  3  4 
12  3  4 


5  6  7 
5  6  7 
5  6  7 
5  6  7 

5  6  7 
5  6  7 
5  6  7 
5  6  7 
5  6  7 

5  6  7 
5  6  7 
5  6  7 
5  6  7 

5  6  7 
5  6  7 
5  6  7 


B.  14  To  what  extent  have  you  been  able  to  successfully  achieve  the  following  outcomes  when 
faced  with  potential  sources  of  instability  in  your  program? 

Not  at  All  Somewhat  Very  Successful 

Avoid  program  instability  l  2  3  4  5  6  7 

before  it  occurs 

Mitigate  the  negative  effects  of  1  2  3  4  5  6  7 

program  instability  after  it  has  occurred 


Section  C:  Program  Schedule  and  Schedule  Planning  Information 

C .  1  Please  estimate  the  date  when  each  of  the  following  key  project/product  milestones  were 
initially  planned  and  are  currently  estimated  to  be  accomplished  for  this  project.  Please 
estimate  if  exact  dates  are  unknown.  Please  provide  the  month  and  year  (e.g.,  8/93) 
or  indicate  not  applicable  in  each  block. 


Key  Program  Milestones  (or  Equivalent) 

Current  Plan 
or  Actual 
Dates 

Initial 

Plan 

Dates 

Start  of  Project  Planning  for  this  Specific  Project 

Mission  Need  Statement  Approved 

Operational  Requirements  Document  Approved 

Project/Product  Acquisition  Start  (Milestone  I)  or  equivalent 

Start  of  the  Demonstration/Validation  Phase  (Phase  II)  or  its  equivalent 

Start  of  the  Eng/Manuf  Development  (Phase  III)  or  its  equivalent 

Start  of  the  Production  Phase  (Phase  IV)  or  its  equivalent 

The  Delivery  of  the  First  Production  Item 

The  Initial  Operational  Capability  Date 

Projected  Develivery  Date  of  Final  Production  Unit 

C  .2  Please  estimate  the  total  change  in  this  project’s  schedule  using  the  initial  project  plan  as  the 
baseline.  Calculate  this  as  die  difference  between  your  current  schedule  and  the  initial 
project  schedule  for  delivery  of  the  first  production  item  (or  other  appropriate  milestone). 
Treat  increases  beyond  original  plan  as  a  positive  change  in  schedule. 

Total  program  schedule  change  _ _ (+  /  -  months  to-date) 

Now,  please  estimate  the  percentage  of  this  project’s  schedule  change  (calculated  from 
above)  that  can  be  attributed  to  unexpected  changes  in  your  requested  annual  funding  levels 
(funding  instability),  changes  in  the  program’s  technical  requirements,  technical  problems, 
or  other  reasons,  respectively.  The  sum  of  the  proportions  should  equal  100%. 

%  of  schedule  change  due  to  funding  instability  _ (%) 

%  of  schedule  change  due  to  requirements  changes  _ (%) 

%  of  schedule  change  due  to  technical  problems  _ (%) 

%  of  schedule  change  due  to  other  reasons  _ (%)  (please  specify:) _ 

100%  _ 


Comments: 


C.3  Briefly  state  how  you  believe  it  was  determined  when  to  start  the  development  of  this 
project  or  product. 


C.4 


Briefly  state  what  factors  were  considered  important  in  setting  the  target  Initial  Operational 
Capability  (IOC)  date. 


C.5  Please  indicate  the  extent  to  which  the  following  factors  helped  determine  the  starting 


date  for  the  development  of  this  project: 

No 

Impact 

User’s  Desires  l  2 

3 

Some 

Impact 

4 

5 

6 

Defining 

Impact 

7 

Service  Leadership  Desires 

1  2 

3 

4 

5 

6 

7 

Service  Planning  Process 

1  2 

3 

4 

5 

6 

7 

Expected  Development  Funding  Availability 

1  2 

3 

4 

5 

6 

7 

Expected  Production  Funding  Availability 

1  2 

3 

4 

5 

6 

7 

Technology  Development 

1  2 

3 

4 

5 

6 

7 

Engineering  Development 

1  2 

3 

4 

5 

6 

7 

Manufacturing  Process  Development 

1  2 

3 

4 

5 

6 

7 

Testing  Requirements 

1  2 

3 

4 

5 

6 

7 

Support  Requirements 

1  2 

3 

4 

5 

6 

7 

Dependence  on  another  program 

1  2 

3 

4 

5 

6 

7 

Other 

1  2 

3 

4 

5 

6 

7 

C.6  Please  indicate  the  extent  to  which  you  believe  the  following  factors  helped  determine  the 
length  of  the  government’s  initial  schedule  for  this  project: 


No 

Some 

Defining 

Impact 

Impact 

Impact 

User’s  Desired  Schedule 

1 

2 

3 

4 

5 

6 

7 

Service  Leadership  Desired  Schedule 

1 

2 

3 

4 

5 

6 

7 

Service  Planning  Process 

1 

2 

3 

4 

5 

6 

7 

Expected  Development  Funding  Availability 

1 

2 

3 

4 

5 

6 

7 

Expected  Production  Funding  Availability 

1 

2 

3 

4 

5 

6 

7 

Technology  Development 

1 

2 

3 

4 

5 

6 

7 

Engineering  Requirements 

1 

2 

3 

4 

5 

6 

7 

Manufacturing  Process  Development 

1 

2 

3 

4 

5 

6 

7  • 

Testing  Requirements 

1 

2 

3 

4 

5 

6 

7 

Support  Requirements 

1 

2 

3 

4 

5 

6 

7 

Dependence  on  another  program 

1 

2 

3 

4 

5 

6 

7 

Other. 

1 

2 

3 

4 

5 

6 

7 

C.7  Please  indicate  the  extent  to  which  you  believe  the  following  tools  or  sources  of 

information  were  used  in  the  development  of  the  government’s  initial  schedule  for 


this  project : 

Not 

Occasionally 

Extensively 

Used 

Used 

Used 

Historical  similar-program  data 

1 

2 

3 

4 

5 

6 

7 

Parametric  modeling  or  estimation 

1 

2 

3 

4 

5 

6 

7 

Expert  judgment 

1 

2 

3 

4 

5 

6 

7 

Bottom  up  schedule  development 

1 

2 

3 

4 

5 

6 

7 

Expected  development  funding  availability 

1 

2 

3 

4 

5 

6 

7 

Expected  production  funding  availability 

1 

2 

3 

4 

5 

6 

7 

Comparable  commercial  development 

1 

2 

3 

4 

5 

6 

7 

Contractor  proposed  schedule 

1 

2 

3 

4 

5 

6 

7 

Other 

1 

2 

3 

4 

5 

6 

7 

C.8  In  your  opinion,  what  was  the  level  of  influence  of  the  following  organizations  in 


developing  the  initial  government  schedule  for  this  project? 

Not  Some 

involved  Influence 

Users  l  2  3  4 

5 

6 

Sole 

Influence 

7 

Program  Office 

1 

2 

3 

4 

5 

6 

7 

FFRDC  (Mitre,  Aerospace,  etc.) 

1 

2 

3 

4 

5 

6 

7 

PEO 

l 

2 

3 

4 

5 

6 

7 

Service  Leadership 

1 

2 

3 

4 

5 

6 

7 

Service  Acquisition  Community 

1 

2 

3 

4 

5 

6 

7 

Service  Cost  Analysis  Group 

1 

2 

3 

4 

5 

6 

7 

Service  Planning  Community 

1 

2 

3 

4 

5 

6 

7 

Service  Budget  Community 

1 

2 

3 

4 

5 

6 

7 

DoD  Leadership 

1 

2 

3 

4 

5 

6 

7 

DoD  Acquisition  Community 

1 

2 

3 

4 

5 

6 

7 

DoD  Budget  Community 

1 

2 

3 

4 

5 

6 

7 

DoD  Planning  Community 

1 

2 

3 

4 

5 

6 

7 

Potential  Contractors) 

l 

2 

3 

4 

5 

6 

7 

Potential  Subcontractors 

l 

2 

3 

4 

5 

6 

7 

Other: 

1 

2 

3 

4 

5 

6 

7 

C .  9  Did  the  Governments,  through  its  RFP  or  other  means,  specify  an  expected  project 
schedule  to  the  contractors? 

□  No.  □  Yes. 


C .  1 0  Please  indicate  the  percentage  by  which  the  winning  contractor’s  schedule  at  the  time  of 
selection  differed  from  the  government’s  proposed/planned  schedule  in  the  RFP  or  the 
government’s  plan.  (Treat  times  longer  than  the  Government  plan  as  positive). 

_ (+/-  %) 

Were  there  other  proposals  submitted  with  different  schedules  than  the  Government  plan? 

□  No  □  Yes  -  If  yes  □  Longer  and/or  □  Shorter 


C .  1 1  Was  development  time  a  significant  evaluation  criterion  during  the  source  selection? 

Not  Somewhat  Very 

Important  Important  Important 

1  2  3  4  5  6  7 


C.12  If  this  program  were  not  constrained  by  resources  (money,  people,  equipment),  at  what 
time  would  this  project’s  first  production  item  be  desired  by  the  Air  Force? 


□  ASAP  □  Date: 


Reason  For  This  Date: 


C.13  Assume  the  contractor  for  this  project  submitted  a  proposal  today  to  shorten  the  remaining 
development  time  to  first  production  item  by  25%  with  the  same  amount  of  total  development 
funding.  In  your  opinion  how  difficult  would  it  be  to  re-plan  the  government’s  aspects  of  the 
project  to  allow  this  schedule  reduction  to  occur?  (25%  of  remaining  development  schedule  = 


months) 

Not 

Somewhat 

Very 

Difficult 

Difficult 

Difficult 

1  2 

3  4  5 

6  7 

To  help  us  further  understand  the  issues  associated  with  shortening  program  schedules  on 
this  specific  project,  please  rate  the  following  items  as  to  how  they  would  impact  the  replanning 
and  potential  approval  of  this  project’s  schedule  change  for  the  above  scenerio. 


Not  A  A  Significant  The  Critical 

Factor  Factor  Factor 


Obtaining  User  Approval 
Obtaining  Service  Approval 
Obtaining  DoD  Approval 


1  2 
1  2 
1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


Changing  The  Development  Funding  Profile 
Changing  The  Production  Funding  Profile 
Acceptance  Of  Program  Risk 


1  2 
1  2 
1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


Contractor’s  Ability  To  Perform  Tasks  Within  Schedule  1  2 

Contractor’s  Ability  To  Perform  Tasks  Within  Budget  1  2 

Contractor’s  Ability  To  Meet  Performance  Requirements  1  2 

Contractor’s  Ability  To  Meet  Maintenance  Requirements  1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


Changing  The  Testing  Program 
Changing  The  Personnel/Training  Planning 
Changing  The  Logistics  Planning 
Integration/Coordination  With  Other  Programs 
Other: _ 


1  2 
1  2 
1  2 
1  2 
1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


In  the  above  scenerio  how  long  do  you  estimate  it  would  take  to  approve  the  reduction  in  schedule 
and  what  is  your  estimate  of  the  probability  of  approval? 

Time  to  Approve _ (Months)  Probability  of  Approval  _ (0-100%) 

What  would  be  the  primary  considerations  for  the  approval  decision? 


C.14  Assume  that  before  the  contract  award  for  this  project,  the  contractor  submitted  a 
proposal  with  a  schedule  that  was  different  than  the  Government’s  planned  schedule  by  the 
indicated  percentages.  How  difficult  would  it  be  to  accommodate  that  schedule  into  the 
government’s  project  plan  considering  all  factors? 

Not  Somewhat  Very 

Months  Difficult  Piffic.uil  Difficult 


+25%  +. 

+10%  +. 
-10% 

-25%  - . 

-50% 


C.  15  To  help  us  further  understand  the  problems  associated  with  shortening  program  schedules, 

assume  that  before  the  contract  award  for  this  project,  the  contractor  submitted  a  proposal 
with  a  schedule  that  was  25%  shorter  than  the  Government’s  planned  schedule  (with  the  same  total 
amount  of  development  funding).  Please  rate  the  following  items  as  to  how  they  would  effect  the 
replanning  of  this  project’s  schedule  for  the  above  scenerio. 


Not  A 

Factor 


A  Significant 

Factor 


The  Critical 
Factor 


Obtaining  User  Approval 
Obtaining  Service  Approval 
Obtaining  DoD  Approval 

Changing  The  Development  Funding  Profile 
Changing  The  Production  Funding  Profile 
Acceptance  Of  Program  Risk 


1  2 
1  2 
1  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


6  7 

6  7 

6  7 


Contractor’s  Ability  To  Perform  Tasks  Within  Schedule  1  2 

Contractor’s  Ability  To  Perform  Tasks  Within  Budget  1  2 

Contractor’s  Ability  To  Meet  Performance  Requirements  1  2 

Contractor’s  Ability  To  Meet  Maintenance  Requirements  1  2 


3  4 

3  4 

3  4 

3  4 


5  6  7 
5  6  7 
5  6  7 
5  6  7 


Changing  The  Testing  Program 
Changing  The  Personnel/Training  Planning 
Changing  The  Logistics  Planning 
Integration/Coordination  With  Other  Programs 
Other: _ _ _ 


12  3  4 
12  3  4 
12  3  4 
12  3  4 
12  3  4 


5  6  7 
5  6  7 
5  6  7 
5  6  7 
5  6  7 


In  the  above  scenerio  how  long  do  you  estimate  it  would  take  to  approve  the  reduction  in  schedule 
and  what  is  your  estimate  of  the  probability  of  approval? 


Time  to  Approve _ (Months)  Probability  of  Approval  _ (0- 100%) 


What  would  be  the  primary  considerations  for  the  approval  decision: 


C.  16  Assume  that  it  was  decided  at  the  time  of  program  initiation  that  the  projected  schedule  for 
this  project  to  first  production  item  should  be  shortened  by  25%.  In  your  opinion,  how  difficult  would  it 
have  been  to  plan  the  government’s  aspects  of  the  project  to  allow  this  schedule  reduction  to  occur? 


(25%  of  the  total  development  schedule  = 

months') 

Not 

Somewhat 

Very 

Difficult 

Difficult 

Difficult 

1  2  3 

4  5  6 

7 

To  help  us  further  understand  the  problems  associated  with  shortening  program  schedules  for  the 
above  scenerio,  please  rate  the  following  items  as  to  how  they  would  effect  die  planning  of  this 


project’s  schedule. 

Not  A 

A 

Significant 

The  Critical 

Factor 

Factor 

Factor 

Obtaining  User  Approval 

1 

2 

3 

4 

5 

6 

7 

Obtaining  Service  Approval 

1 

2 

3 

4 

5 

6 

7 

Obtaining  DoD  Approval 

1 

2 

3 

4 

5 

6 

7 

Changing  The  Development  Funding  Profile 

1 

2 

3 

4 

5 

6 

7 

Changing  The  Production  Funding  Profile 

1 

2 

3 

4 

5 

6 

7 

Acceptance  Of  Program  Risk 

1 

2 

3 

4 

5 

6 

7 

Contractor’s  Ability  To  Perform  Tasks  Within  Schedule 

1 

2 

3 

4 

5 

6 

7 

Contractor’s  Ability  To  Perform  Tasks  Within  Budget 

1 

2 

3 

4 

5 

6 

7 

Contractor’s  Ability  To  Meet  Performance  Requirements 

1 

2 

3 

4 

5 

6 

7 

Contractor’s  Ability  To  Meet  Maintenance  Requirements 

1 

2 

3 

4 

5 

6 

7 

Changing  The  Testing  Program 

1 

2 

3 

4 

5 

6 

7 

Changing  The  Personnel/Training  Planning 

1 

2 

3 

4 

5 

6 

7 

Changing  The  Logistics  Planning 

1 

2 

3 

4 

5 

6 

7 

Integration/Coordination  With  Other  Programs 

1 

2 

3 

4 

5 

6 

7 

Other: 

1 

2 

3 

4 

5 

6 

7 

C.17  Assuming  no  resource  constraints,  how  long  do  you  estimate  that  it  could  have  taken  to  go 
from  project  start  to  first  production  item  knowing  what  you  know  at  this  point  in  die 
project. 

_ Months  _ Months  Different  from  Current  Plan 


C.18  In  your  opinion  what  would  have  been  the  optimal  development  schedule  to  first 
production  item  for  this  project?  (considering  all  aspects  of  cost,  schedule,  and 
performance  and  other  factors  you  feel  are  important  for  this  program). 

_ Months  _ Months  Different  from  Current  Plan 


Section  D:  Personal  Information 


D .  1  For  how  long  have  you  worked  on  this  project?  - (months) 

D.2  Including  yourself,  how  many  people  have  filled  your  position  on  this  project  since  it  was 
begun  (all  program  phases  to  date)?  - (#  of  people) 

D .  3  On  the  whole,  do  you  consider  yourself  more  of  an  operational  officer  or  an  acquisition 
officer? 

□  Operational  □  Acquisition 


D.4  Please  indicate  the  significant  acquisition 
completed: 

□  Sys  100  or  equivalent 

□  Sys  200  or  equivalent 

□  Sys  400  or  equivalent 

□  DoD  Systems  Engineering  Management  Course 

□  DSMC  Program  Management  Course  (PMC) 


courses  or  training  that  you  have  taken  or 

□  MBA/Management  Masters  Degree 

□  Masters  Degree  in _ _ 

D  Product  development  specific  course  at  a  business 

school  or  university 

□  Others)  _ _ 


D.5  Please  rank  from  one  to  four  the  following  items  in  terms  of  your  perception  of  their 
importance  in  the  development  of  this  project  (“1”  indicates  the  most  important  and  “4”  the  least 
importance): 

_ Lowering  Project  Acquisition  Cost  _ Raising  the  System  Performance 

_ Shortening  Project  Development  Schedule  _ Lowering  the  Opertional/Maintenance  Costs 


D.6  Please  indicate  the  extent  that  you  feel  meeting  or  beating  the  cost,  schedule, 

performance,  and  maintenance  requirements  for  this  program  (as  opposed  to  other  areas 
involved  in  your  job)  influences  your  performance  rating  and/or  potential  for  promotion: 


No 

Some 

Very  Larj 

Impact 

Impad 

Impact 

Not  Meeting  the  goals 

1 

2 

3 

4 

5 

6  7 

Meeting  the  goals 

1 

2 

3 

4 

5 

6  7 

Beating  the  goals 

1 

2 

3 

4 

5 

6  7 

D.7  Please  rank  the  following  items  in  terms  of  their  importance  to  your  personal  performance 
rating  and  your  potential  for  promotion  (“1”  indicates  the  item  with  the  greatest  importance 
to  your  potential  for  promotion,  “8”  indicates  the  item  with  the  least  importance  to  your 
potential  for  promotion): 


Completing  project  on  budget 
Completing  project  under  budget 
Completing  project  on  schedule 
Completing  project  ahead  of  schedule 


Meeting  technical  performance  requirements 
Exceeding  technical  performance  requirements 
Meeting  operability/maintainability  requirements 
Exceeding  operability/maintainability  requirements 


Comments: 


D .  8  So  that  I  may  contact  you  if  I  have  questions  on  your  survey  responses,  please  provide  your 
name,  address  and  phone  number 

Name:  _ _ _ _ 

Mailing  Address:  _ _ _ _ 


Tel: 


□  Yes,  I  would  like  a  summary  of  the  survey  results. 

If  you  have  any  comments  or  recommendations  about  the  issues  in  this  survey  feel 
free  to  write  them  in  the  space  provided. 

Comments: 


Thank  you  very  much  for  your  time  and  assistance!!!! 


N° 
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(Confidential  Questionnaire) 


Massachusetts  Institute  of  Technology 
Lean  Aircraft  Initiative 


Instructions 


This  survey  is  part  of  the  Lean  Aircraft  Initiative  (LAI),  on-going  research  program  involving  the 
U.S.  Air  Force  and  a  number  of  other  government  agencies  and  departments,  several  firms  in  the 
U.S.  defense  aerospace  sector,  and  academia.  The  LAI  is  investigating  the  application  of  “lean” 
management  practices  to  a  wide  range  of  activities  throughout  the  defense  aerospace  industry.  This 
survey  is  designed  specifically  to  assess  the  impact  of  various  attributes  of  the  defense  acquisition 
environment  on  the  application  of  lean  management  practices  at  the  program  level.  If  your 
customer  is  a  U.S.  government  agency  or  department,  a  similar  survey  may  have  already  been 
completed  by  your  government  customer’s  program  office. 

Your  cooperation  is  vital  to  the  success  of  this  study!  To  ensure  accuracy  and  consistency  of  the 
results,  we  would  prefer  that  the  program  manager  fill  out  as  much  of  the  survey  as  possible. 
Please  answer  the  following  questions  as  they  apply  specifically  to  your 
individual  program  or  project  (your  current  contract).  The  following  examples  may 
help  you  in  determining  how  you  should  respond  to  the  questions: 

Example  #1:  You  are  on  contract  with  your  customer  to  produce  a  subsystem  for  a  larger 
system.  Your  survey  responses  should  characterize  the  work  on  your  specific  subsystem  only. 
Example  #2:  Your  firm  has  several  contracts  with  your  customer  associated  with  an  existing 
system  (encompassing  different  phases  in  its  product  life  cycle,  or  different  types  of  work  within  a 
given  program  phase).  Your  survey  responses  should  characterize  the  work  associated  with  one 
specific  contract  only.  Separate  surveys  may  be  completed  for  the  other  contracts. 

Example  #3:  Your  firm  has  separate  contracts  associated  with  this  product,  each  with  a  different 
customer.  Your  survey  responses  should  characterize  the  work  associated  with  one  specific 
contract  only.  Separate  surveys  may  be  completed  for  the  other  contracts. 

Please  be  candid  and  honest  in  your  responses.  We  understand  that  you  may  have  concerns  about 
confidentiality.  Several  measures  have  been  taken  to  ensure  that  your  responses  will  remain 
confidential.  Nobody  but  the  researchers  named  at  the  end  of  the  survey  will  have  access  to  the 
information  requested  in  this  survey.  All  analysis  of  the  survey  data  will  be  presented  in  the  form 
of  aggregated  statistics.  No  individuals  or  individual  programs  will  be  identified  in  the  analysis  or 
the  reporting  of  the  responses.  We  understand  that  the  success  of  any  research  depends  upon  the 
quality  of  the  information  on  which  it  is  based,  and  we  take  seriously  our  responsibility  to  ensure 
that  any  information  you  entrust  to  us  will  be  protected  with  the  same  care  that  you  would  use 
yourself. 

When  you  have  completed  the  survey,  please  enclose  it  in  the  accompanying  envelope  and  return  it 
directly  to  the  researchers.  We  would  prefer  that  you  return  the  completed  survey  to  us  as  soon  as 
possible,  but  to  ensure  timely  and  complete  reporting,  please  return  it  by  the  date  indicated  in  the 
mailing  materials.  If  you  would  like  feedback  on  the  results  of  the  survey,  please  indicate  it  in  the 
space  provided  at  the  end  of  the  survey.  Thank  you  for  your  help  in  this  important  project. 


Program  Name: 
Company: 


All  information  gathered  through  this  survey  is  considered 
confidential.  No  attribution  of  specific  data  or  research  results 
to  this  program  will  be  made.  All  results  will  be  presented  in 

the  form  of  aggregate  statistics. 


Section  A:  Program  Management 

A .  1  What  is  the  DoD  acquisition  category  (ACAT)  designation  for  your  program? 


A .  2  In  what  acquisition  phase  is  your  program  currently? 

□  Concept  Exploration  &  Definition  □  Operation  &  Support 

□  Demonstration  &  Validation  □  Close-out/Cancellation 

□  Engineering  &  Manufacturing  Development  □  Other: _ 

□  Production  &  Deployment 


A .  3  Please  identify  your  customer(s).  Mark  all  that  apply,  but  indicate  your  primary  customer. 

D  U.S.  Air  Force  □  Prime  or  other  contractor 

□  U.S.  Army  □  Foreign  customer  (commercial  sale) 

□  U.S.  Navy  □  Foreign  customer  (FMS  sale) 

□  Other  U.S.  government  agency  □  Other:  _ _ 


A  .4  Was  this  program  awarded  on  the  basis  of  a  sole  source  or  a  competitive  bidding  process? 
□  Sole  Source  □  Competitive  bid  (number  of  competitors: _ ) 


A. 5  What  is  your  total  program  length  (measured  as  the  elapsed  time  between  the  program’s 
start  and  the  actual  or  planned  delivery  of  the  first  deliverable  item) _ (months) 


□  Aircraft  (airframe  and  mechanical  systems) 

□  Aircraft  (avionics  and  electronic  systems) 

□  Aircraft  (propulsion) 

□  Spacecraft  or  launch  system 


type  of  system  your  program  is  developing? 

Electronic  system  (non-aircraft) 

□  Missile  or  munitions 

□  Software-dominated  system 

□  Other: _ 


A .  6  Which  of  the  following  categories  best  describes  the 

□ 


A. 7  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
product  technologies  required  by  your  program,  relative  to  existing  systems: 

D  Revolutionary  new  core  technologies  or  concept. 

□  New  generation  of  product  architecture  or  platform. 

□  Incremental  improvements  to  an  existing  generation  of  product  architecture  or  platform. 

□  Little  or  no  change  to  the  existing  product  technology. 


A. 8  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
process  (manufacturing)  technologies  required  by  your  program,  relative  to  existing 
processes: 

□  Revolutionary  new  core  process  technology  or  concept. 

□  New  generation  of  existing  processes. 

□  Incremental  improvements  to  an  existing  generation  of  process  technologies. 

□  Little  or  no  change  to  the  existing  process  technology. 


A. 9  Please  estimate  the  extent  of  the  technological  advance  required  in  the  product  technology 
component(s)  that  are  considered  critical  to  fulfilling  this  program’s  technical  objectives: 

All  Non-  Some  All  New 

Developmental  Development  Development 

Items  Required  Required 

1  2  3  4  5  6  7 


A .  1 0  What  percentage  of  your  product’s  cost  to  date  is  related  to  software? _ (%) 

A.  11  To  what  extent  do  commercial  markets  currently  exist  for  the  key  or  enabling  product 
technologies  being  used  in  this  program? 

No  Some  Many 

Markets  Markets  Markets 

1  2  3  4  5  6  7 

A.  12  As  far  as  you  are  aware,  how  many  other  active,  existing  programs  (either  within  your  own 
firm  or  involving  your  competitors)  are  developing  systems  or  technologies  that  will 
compete  to  fulfill  the  same  or  similar  operational  objectives  as  your  program? 

Within  the  U.S.  _  Outside  of  the  U.S.  _ 


A.  13  For  how  long,  and  during  how  many  program  phases  (e.g.,  DEM/VAL,  EMD,  etc.)  have 
you  been  the  manager  of  this  program? 

_ (years)  _ (months)  _ (total  number  of  program  phases) 


A.  14  Including  yourself,  how  many  different  program  managers  have  led  this  program  at  your 
firm  since  it  was  begun  (all  program  phases  to  date)? _ 


A.  15  How  many  other  programs  involving  technical  or  mission  objectives  similar  to  this 
program’s  have  you  managed  prior  to  working  on  this  specific  program? _ 

A .  1 6  How  many  people  currently  work  at  your  firm  on  this  program  (this  program  phase  only)? 

Employees  of  your  firm  working  full-time  on  this  program  _ 

Employees  of  your  firm  working  part-time  on  this  program  _ _ 

Contractor  support  (subcontractors,  vendors,  temporary  employees)  _ 

Total  full-time  equivalents  (FIE)  _ 


A.  17  What  percentage  of  this  program’s  current  staff  began  working  on  this  program  at  it’s  start 
and  at  the  start  of  this  program  phase,  respectively,  and  are  still  working  on  this  program? 

This  program  phase  only _ (%)  The  entire  program _ (%) 


A.  18  Please  estimate  the  percentage  of  this  program’s  current  staff  who  worked  on  at  least  one 
other  program  that  produced  a  system  with  similar  technical  or  operational  characteristics: 

_ .(%) 


A.  19  To  what  extent  would  you  say  that  your  program’s  progress  in  meeting  its  overall 
objectives  during  this  program  phase  has  been  affected  by  program  instability  (defined  as 
unplanned  or  unanticipated  changes  in  your  requested  annual  budget  or  other  critical 
resources,  program  schedule,  or  definition  of  work)? 

No  Effect  Some  Effect  Major  Effect 

1  2  3  4  5  6  7 


A. 20  To  what  extent  do  you  think  the  following  factors  have  introduced  instability  into  your 
program’s  planning  and  execution? 


Changes  in  annual  budget  allocations 

Changes  in  annual  production  or  block  buy  size 

Changes  in  user  requirements 

Unanticipated  technical  challenges  originating 
within  this  program 

Problems  resulting  from  technical  challenges  in 
other,  related  programs 

Problems  relating  to  cooperation  with 
customer(s),  partners,  or  suppliers 

Program  staffing  turnover  within  your  firm. 

Program  staffing  turnover  in  your  customer’s 
organization 

Problems  resulting  from  poor  supplier 
performance 

Changing  customer  acquisition  priorities 
Long  acquisition  cycle  time 
Other: 


Not  a 
Factor 

Somewhat 

Factor 

a 

The  Primary 
Factor 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

A. 21  How  many  times  was  your  program’s  formal  contract  management  baseline  re-baselined 
(with  customer  approval)  in  each  of  the  following  program  phases? 

Concept  Exploration  _ 

Demonstration/Validation  _ 

Engineering  and  Manufacturing  Development  _ 

Production 


A. 2 2  How  has  the  level  of  profit  (as  a  percentage  of  total  program  cost)  that  was  anticipated  to 
result  from  this  program  when  your  firm  was  awarded  this  contract  been  affected  by 
program  instability? 

Significantly  No  Significantly 

Declined  Effect  Increased 

1  2  3  4  5  6  7 


A. 2 3  To  what  extent  have  you  been  able  to  successfully  achieve  the  following  outcomes  when 


faced  with  potential  sources  of  instability  in  your  program? 

Not 
A-t_ AH 

Somewhat 

Very 

S-UccessfuILy 

Avoid  program  instability  before  it  occurs 

1 

2 

3 

4 

5 

6 

7 

Mitigate  the  negative  effects  of  program 
instability  after  it  has  occurred 

1 

2 

3 

4 

5 

6 

7 

A. 2 4  Please  indicate  the  extent  to  which  you  believe  the  following  practices  have  had  an  impact 
in  helping  to  assure  stability  (i.e.,  avoid  instability)  in  your  program’s  funding, 
requirements,  and/or  planning/schedule: 

Not 

Used 

Not 

Significant 

Somewhat 

Significant 

Very 

Significant 

Customer  involves  the  product’s  users  (if 
different)  in  developing  requirements 

0 

1 

2 

3 

4 

5 

6 

7 

Customer  involves  contractors)  in  developing 
requirements 

0 

1 

2 

3 

4 

5 

6 

7 

Maintain  open  and  frequent  communication  with 
the  customer 

0 

1 

2 

3 

4 

5 

6 

7 

Use  independent  experts  to  assess  program  plan 
and  execution 

0 

1 

2 

3 

4 

5 

6 

7 

Structure  program  around  a  short  overall 
acquisition  time 

0 

1 

2 

3 

4 

5 

6 

7 

Base  design  on  incremental  technology  steps 

0 

1 

2 

3 

4 

5 

6 

7 

Use  off-the-shelf  or  non-developmental 
technologies  in  the  product  design 

0 

1 

2 

3 

4 

5 

6 

7 

Manage  high-risk  development  items  off  the 
critical  program  path 

0 

1 

2 

3 

4 

5 

6 

7 

Schedule  interdependent  technical  activities  to 
occur  within  the  same  budget  cycle 

0 

1 

2 

3 

4 

5 

6 

7 

Validate  designs  using  prototypes 

0 

1 

2 

3 

4 

5 

6 

7 

Validate  designs  using  simulation  and  modeling 

0 

1 

2 

3 

4 

5 

6 

7 

Use  multi-year  procurement 

0 

1 

2 

3 

4 

5 

6 

7 

Maintain  contract  flexibility  (e.g.,  priced 
options,  umbrella  contracts) 

0 

1 

2 

3 

4 

5 

6 

7 

Establish  rigid  contracts  with  stiff  penalties  for 
unplanned  changes 

0 

1 

2 

3 

4 

5 

6 

7 

Aggressively  advocate  support  for  your  program 
to  senior  officials  in  customer  organization(s) 
(e.g.,  the  service(s),  DoD,  Congress,  etc.) 

0 

1 

2 

3 

4 

5 

6 

7 

A. 2 5  Please  indicate  the  extent  to  which  you  have  found  the  following  management  practices  or 
technology  tools  effective  at  responding  to  or  mitigating  the  negative  impact  of 
unplanned  changes  in  budget,  requirements,  and/or  schedule  to  your  program  after  they 
have  occurred: 


Budget  management  reserve  in  the  program  plan 

Use  unused  funding  from  prior  years 

Use  your  firm’s  resources  to  meet  temporary 
customer-originated  funding  shortfalls 

Use  temporary  or  contractor  personnel  for 
staffing 

Use  personnel  from  other  programs  on  an  as- 
needed  basis 

Structure  work  around  integrated  product  teams 

Use  computer-aided  scheduling  and  task 
breakdown  planning 

Use  3-D  computer  modeling  for  design 

Use  rule-based  or  knowledge-based  design  tools 

Use  rapid  prototyping  tools  (e.g., 
stereolithography) 

Use  computer-aided  manufacturing 

Use  flexible  or  tool-less  assembly  techniques 

Use  manufacturing  process  modeling  and 
simulation  tools 

Develop  “fallback”  alternatives  to  key 
subsystems  using  parallel  IR&D  projects 

In-source  work  from  or  out-source  work  to 
suppliers  to  meet  fluctuation  in  demand 

Other: _ _ 


Not 

Used 

Not 

Effective 

'  Somewhat 
Effective 

Very 

Effective 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

A. 2 6  By  what  percentage  can  your  annual  budget  and/or  production  rate  vary  before  a  significant 
amount  of  effort  (beyond  that  typically  required  for  normal  program  control  and  execution) 
is  required  to  re-plan  the  program’s  work  schedule  or  objectives? 

Change  in  annual  budget _ (%)  Change  in  annual  production  rate _ (%) 


A.  27  By  how  much  has  the  number  of  units  or  systems  that  are  planned  to  be  procured  (the  “total 
buy”)  over  the  life  of  this  program  changed  since  the  program’s  start,  and  how  has  this 
change  directly  impacted  unit  (“fly-away”)  price,  if  at  all? 

Production  quantity  (plan  at  program  start)  _ 

Production  quantity  (current  plan)  _ 

%  Change  in  unit  price  _ (+  /.%) 

Quantity  produced  to  date  _ 


A .  2  8  How  many  times  has  the  total  number  of  units  that  are  planned  to  be  procured  over  the  life 
of  this  program  been  changed  in  each  of  the  following  program  phases? 

Concept  Exploration  phase  _ 

Demonstration^/ alidation  phase  _ 

Engineering  and  Manufacturing  Development  phase  _ 

Production  phase  _ 


A. 2 9  Please  estimate  the  level  of  effort  in  your  program  (in  FTEs — annual  full-time  equivalent 
staff  or  “man-years”)  that  is  spent  exclusively  planning  your  program’s  schedule  and 
execution  in  anticipation  of  or  as  a  result  of  unplanned  changes  in  funding 
and/or  other  key  resources: 

Planning  prior  to  the  annual  appropriations  (“budget  exercises”)  _ (FTE) 

Re-planning  in  response  to  actual  budget  changes  _ (FTE) 


A. 30  Please  indicate  the  extent  to  which  the  existence  of  planning  and  resource  uncertainty  or 
instability  in  your  program  may  have  prevented  your  firm  from  undertaking  the  following 
actions  as  they  relate  to  your  program  (N/A  -  not  applicable/not  used): 


Internal  investment  in  basic  technology  research 

Internal  investment  in  next-generation  product 
technology  development 

Internal  investment  in  advanced  information 
technologies 

Internal  investment  in  advanced  manufacturing 
process  technology 

Internal  investment  for  variability  reduction  in 
existing  manufacturing  process  technologies 

Internal  investments  in  factory  floor  layout 
changes  for  manufacturing  flow  improvements 

Employee  training  and  skill  development 

Maintaining  integrated  product  team  (IPT) 
staffing  continuity  throughout  the  product  life 
cycle 

Developing  long-term  agreements  with  your 
suppliers 

Advanced  inventory  and  scheduling 
management  (e.g.,  JIT)  with  suppliers 


N/A 

0 

No 

Impact 

1 

2 

3 

Some 

Impact 

4 

5 

Very  Lar$ 
Impact 

6  7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

A .  3 1  What  is  the  average  lead  time  from  placing  orders  to  your  suppliers  after  source  selection  to 
the  first  delivery  of  a  deliverable  item  to  the  customer  for  the  current  production  lot  for  this 
program? 

“Long  lead”  items _ (months)  All  other  items _ _  (months) 


A. 3 2  What  percentage  of  your  total  program  budget  represents  work  performed  or  materials 
furnished  by  suppliers  or  vendors? _ (%) 


A. 3 3  What  percentage  of  your  program’s  purchases  from  suppliers  represent  critical  parts, 
components,  materials,  or  processes  for  which  switching  suppliers  is  either  not  possible  or 
is  extremely  costly  in  terms  of  maintaining  budget,  schedule,  or  quality  goals? 
_ .(%) 

What  percentage  of  your  program’s  total  number  of  suppliers  supplies  your  program  with 
these  types  of  critical  items? _ (%) 

What  percentage  of  your  program’s  suppliers  of  critical  items  (requested  above) 
present  a  realistic  potential  threat  of  electing  not  to  compete  for  future  contracts  relating  to 
your  program,  citing  such  reasons  as  the  high  cost  to  them  of  program  instability,  uncertain 
or  highly  variable  demand  in  the  defense  sector,  costly  defense-unique  requirements,  or 
excessive  contractual  oversight? _ (0- 100%) 


A. 3 4  Please  indicate  the  extent  to  which  program  instability  has  limited  your  ability  to  establish 
or  maintain  the  following  types  of  relationships  with  your  suppliers  (N/A  -  not 


applicable/not  used): 


Involve  suppliers  in  IPTs  throughout  the 
product  life  cycle 

Involve  suppliers  in  the  definition  of  the 
product’s  technical  specifications 

Delegate  detailed  design  work  on  major 
subsystems  to  suppliers 

Integrate  key  suppliers  into  capacity-planning 
and  scheduling  systems 

Achieve  consistent  on-time  delivery  of  materials 
from  suppliers 

Maintain  reliable  production  base  of  suppliers 
with  defense-unique  capabilities 

Maintain  reliable  production  base  of  suppliers 
that  are  process-certified  or  meet  strict  quality 
levels 


N/A 

No 

ImDact 

Some 

Impact 

Very  Larj 
Impact 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

0 

1 

2 

3 

4 

5 

6 

7 

A  .35  Please  estimate  the  priority  placed  on  your  program  in  each  of  the  following  organizations: 


Very  Lowest 

Average 

Very  Highest 

Priority 

Priority 

Priority 

Your  customer 

i 

2 

3 

4 

5 

6 

7 

Your  division 

i 

2 

3 

4 

5 

6 

7 

Your  corporation 

i 

2 

3 

4 

5 

6 

7 

Your  suppliers  of  critical  parts,  components, 
materials,  or  processes  (see  A.33  above) 

i 

2 

3 

4 

5 

6 

7 

Section  B:  Program  Budget  and  Contracts 


B.l  What  is  your  program’s  current  target  or  actual  budget  expenditure  for  all  program 
expenses  (as  defined  by  your  customer),  in  each  of  the  following  program  phases?  Use 
actual  dollars  and  indicate  the  reference  year. 


Concept  Evaluation 

DEM/VAL 

EMD 

Production 


($milhon)  FY(s) _  Target  /  Actual  (circle  one) 

(SmUlion)  FY(s) _  Target  /  Actual  (circle  one) 

(Smfllion)  FY(s) _  Target  /  Actual  (circle  one) 

(Smillion)  FY(s) _  Target  /  Actual  (circle  one) 


B .  2  Please  estimate  the  change  to  date  in  your  program’s  cost  (from  all  sources)  using  the  initial 
program  plan  as  a  baseline.  Calculate  this  as  the  percentage  change  in  actual  cost  compared 
with  the  initially-planned  cost  for  this  point  in  your  program’s  progress,  using  actual-year 
dollars.  Treat  increases  beyond  original  plan  as  a  positive  change  in  cost. 

Total  program  cost  change _  (+/-%  to-date)  (indicate  baseline  year) _ 

Now,  estimate  the  percentage  of  your  program’s  cost  change  to  date  (calculated  above) 
that  can  be  directly  attributable  to  unplanned  changes  in  your  requested  annual  funding 
levels  (funding  instability),  changes  in  the  program’s  technical  requirements,  technical 
problems,  or  other  reasons,  respectively.  The  sum  of  the  proportions  should  equal  100%. 

%  of  cost  change  due  to  funding  instability  _  _ (%) 

%  of  cost  change  due  to  requirements  changes  _ (%) 

%  of  cost  change  due  to  technical  problems  _ (%) 

%  of  cost  change  due  to  other  reasons  (%)  (please  specify:) _ 

100%  _ 


B .  3  What  type  of  contract  does  your  program  have  with  your  customer  in  this  program  phase? 


□  Firm  Fixed  Price  (FFP) 

□  Fixed  Price  with  Economic  Price  Adjustment 

□  Fixed  Price  Incentive  (FPI) 

□  Fixed  Price  with  Redetermination  (FPR) 

□  Cost  Plus  Award  Fee  (CPAF) 

□  Cost  Plus  Incentive  Fee  (CPEF) 


□  Cost  Plus  Fixed  Fee  (CPFF) 

□  Cost  and  Cost  Sharing  (C/CS) 

□  Letter  Contract 

□  Indefinite  Delivery/Indefinite  Quantity 

□  Other _ 


B.4  In  your  current  contract,  what  percentage  of  program  cost  over-runs  or  under-runs  does 
your  firm  share  with  your  customer  near  the  target  price? 

Your  firm’s  share  of  cost  over-runs  _ (%) 

Your  firm’s  share  of  cost  under-runs  _ (%) 


B .  5  What  is  your  program’s  total  budget  level  (as  defined  by  your  customer)  for  this  year? 
_ (Smillion) 


B .  6  By  what  percentage  did/does  your  program’s  annual  budget  allocation  from  your  customer 
differ  from  that  requested  in  your  program  plan,  if  any?  Use  +  /  -  to  indicate  increases  or 
decreases  from  plan,  respectively. 

This  year _ (%)  On  average _ (%) 


B  .7  How  many  times  has  this  program  been  re-phased  (a  significantly  new  program  schedule 
developed  by  your  customer)  in  each  of  the  following  program  phases? 

Concept  Exploration  phase  _ 

Demonstration^/  alidation  phase  - 

Engineering  and  Manufacturing  Development  phase  _ 

Production  phase  - 


B.8  How  many  contract  change  orders  (e.g.,  “POOOxxx”  changes)  has  this  program  received 
from  its  customer  in  all  program  phases  to  date?  _ 

Of  the  total  number  of  contract  change  orders  requested  above,  wha‘  percentage  were 
“Incremental  Funding  Changes”? _ (%) 

How  many  of  the  total  number  of  contract  change  orders  requested  above  changed  your 
contract’s  scope/requirements,  deliverables,  funding  (include  all  types  of  funding  changes), 
or  schedule?  List  each  type  of  change  by  the  program  phase  in  which  it  occurred  in  the 
matrix  below.  Please  estimate  if  you  don’t  know  the  exact  number  of  changes  of  each  type. 


Program  Phase 

Scope/ 

Requirements 

Deliverables/ 

Quantity 

Funding 

Schedule 

DEM/VAL 

EMD 

Production 

Total 

B .  9  Please  indicate  the  extent  to  which  your  customer  provides  your  program  with  specific 
financial  or  other  types  of  incentives  to  achieve  the  following  objectives: 

A  Strong  No  A  Strong 


Reduce  total  program  cost  below  target 
Reduce  unit  cost  below  target 
Complete  program  ahead  of  schedule 
Exceed  planned  performance  requirements 
Exceed  reliability  and  maintenance  requirement 


Disincentive 

Incentive 

Incentive 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

B.10  Considering  the  bottom-line  impact  on  your  company  overall,  please  indicate  the  net 
financial  incentive  your  program  has  to  achieve  the  following  objectives: 

A  Strong  No  A  Strong 


Reduce  total  program  cost  below  target 
Reduce  unit  cost  below  target 
Complete  program  ahead  of  schedule 
Exceed  planned  performance  requirements 
Exceed  reliability  and  maintenance  requirement 


Disincentive 

Incentive 

Incentive 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

Section  C:  Program  Schedule  and  Planning 

C .  1  Please  indicate  when  each  of  the  following  key  program  milestones  were  accomplished  or 
when  they  are  scheduled  to  be  accomplished  for  your  program.  Provide  the  month  and 
year  (e.g.,  8/93)  or  quarter  and  year,  as  well  as  indicating  the  status  of  the  milestone. 
Please  estimate  if  exact  dates  are  unknown. 


Key  Program  Milestones  (or  Equivalent) 

wmnsEEEm 

Status 

Program  Start  (Milestone  I  or  equivalent) 

Actual  -  Planned  -  N/A 

Concept  Evaluation 

Actual  -  Planned  -  N/A 

Demonstration/V  alidation 

Actual  -  Planned  -  N/A 

Engmeering/Manufacturing  Development 

Actual  -  Planned  -  N/A 

Production 

Actual  -  Planned  -  N/A 

delivery  of  the  first  production  item 

Actual  -  Planned  -  N/A 

Initial  Operational  Capability 

Actual  -  Planned  -  N/A 

Note:  N/A — Not  Applicable  or  no  date  available. 


C.2  Is  your  program  considered  “fast  track”  (the  schedule  has  officially  been  accelerated  so  as 
to  shorten  the  total  acquisition  cycle)? 

□  No.  □  Yes.  . 


C.3  Please  estimate  the  total  change  in  your  program’s  schedule  using  the  initial  program  plan 
as  the  baseline.  Calculate  this  as  the  difference  between  your  current  schedule  and  the 
initial  program  schedule  for  meeting  specific  milestones  at  this  point  in  your  program  (treat 
increases  beyond  original  plan  as  a  positive  change  in  schedule). 

Total  program  schedule  change  _ (+  /  -  months  to-date) 

Now,  estimate  the  percentage  of  your  program’s  schedule  change  to  date  (calculated 
above)  that  can  be  directly  attributable  to  unexpected  changes  in  your  requested  annual 
funding  levels  (funding  instability),  changes  in  the  program’s  technical  requirements, 
technical  problems,  or  other  reasons,  respectively.  The  sum  of  the  proportions  should 
equal  100%. 

%  of  schedule  change  due  to  funding  instability  (%) 

%  of  schedule  change  due  to  requirements  changes  _ (%) 

%  of  schedule  change  due  to  technical  problems  _ _ (%) 

%  of  schedule  change  due  to  other  reasons  (%)  (please  specify:!  _ 

100%  _ _ • 


C .  4  By  how  many  months  could  you  accelerate  the  scheduled  delivery  of  the  first  deliverable 
item  for  your  program  if  you  faced  no  customer-imposed  schedule  or  resource  constraints? 
_ (Months) 


C.5  What  net  incentive  (taking  into  account  cost,  risk,  and  other  competitive  issues)  does  your 
organization  have  to  bid  a  schedule  different  from  that  proposed  by  the  government? 


A  Strong  No 

Disincentive  Incentive 


A  Strong 
Incentive 


1  2  3  4  5  6  7 


C.6  Please  indicate  the  percentage  by  which  your  proposal  schedule  differed  from  the 
government’s  proposed  schedule  in  the  RFP.  A  negative  value  indicates  a  shorter  schedule 
and  a  positive  value  indicates  a  longer  schedule  than  the  government’s,  respectively. 
_ (%) 


C .  7  Please  indicate  the  extent  to  which  each  of  the  following  defined  the  program  schedule  you 

proposed  in  your  response  to  the  RFP  for  this  program: 

No 

Impact 

Moderate 

Impact 

Sole 

Determinant 

Customer’s  desired  schedule 

1  2 

3  4 

5  6 

7 

Expected  funding  availability 

1  2 

3  4 

5  6 

7 

Historically  similar  programs 

1  2 

3  4 

5  6 

7 

Comparable  commercial  development 

1  2 

3  .4 

5  6 

7 

Expert  judgment/Bottom-up  analysis 

1  2 

3  4 

5  6 

7 

C .  8  Using  the  Commerce  Business  Daily  announcement  for  your  program  as  the  starting  point, 
please  indicate  the  approximate  time  to  accomplish  the  following  program  events.  Please 
indicate  the  change  (if  any)  from  your  customer’s  target  schedule  for  their  completion. 


Program  Event 

Months 

■Syijg 

Pre-Program  Efforts  (IR&D,  Marketing,  Etc.) 

Commerce  Business  Daily  announcement 

0 

Draft  RFP  Release  (If  Applicable) 

Final  RFP  Release 

Proposal  Submittal 

Source  Selection  Announcement 

Contract  Award 

First  Production  Item  or  Product  Delivery 

Initial  Operational  Capability 

C.9  Please  indicate  the  extent  to  which  your  firm  was  able  to  impact  the  following  items 
associated  with  your  customer’s  request  for  proposal  (RFP)  for  this  program  prior  to  its 


release: 

Program  or  product  concept 
Program  schedule 

Overall  program  budget  or  funding  profile 

System  performance  requirements 

Acquisition  strategy  (source  selection  process, 
contract  structure,  etc.) 

Trade-offs  between  performance,  cost,  and 
schedule  in  the  program  plan 


No 

Some 

Very  Larj 

Impact 

Imoact 

Imoact 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

If  you  have  any  comments  or  recommendations  about  the  issues  addressed  by  this  survey,  feel  free 
to  write  them  in  the  space  provided  below: 


□  Yes,  I  would  like  a  summary  of  the  results  of  this  research  sent  to  me. 
Name: 

Mailing  Address: 


Tel: 


survey  or  the  objectives  of  this  research,  feel  free  to  contact 


If  you  have  any  questions  about  this 
either 

Dr.  Eric  Rebentisch 
Lean  Aircraft  Initiative 
Massachusetts  Institute  of  Technology 
77  Massachusetts  Ave.,  Room  33-407 
Cambridge,  MA  02139 
tel.:  (617)  258-7773 
fax:(617)258-7845 
email:  erebenti@mit.edu 


Capt.  Ross  McNutt 
Lean  Aircraft  Initiative 
Massachusetts  Institute  of  Technology 
77  Massachusetts  Ave.,  Room  33-407 
Cambridge,  MA  02139 
tel.:  (617)  258-6594 
fax:  (617)  258-4235 
email:  mcnuttrt@mit.edu 


For  more  information  about  the  Lean  Aircraft  Initiative,  you  may  contact  us  directly  or  visit  the  LAI 
web  page  at:  http://web.mit.edu/ctpid/www/lai/. 


Thank  You  Very  Much!!!! 

All  information  gathered  through  this  survey  is  considered 
confidential.  No  attribution  of  specific  data  or  research  results 
to  this  program  will  be  made.  Ail  results  will  be  presented  in 

the  form  of  aggregate  statistics. 
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Survey  of  the  Impact  of 

the  Defense  Aerospace  Acquisition  Environment 
on  the  Use  of  Lean  Program  Management  Practices 

©  Massachusetts  Institute  of  Technology,  1995 


( Confidential  Questionnaire ) 


Massachusetts  Institute  of  Technology 
Lean  Aircraft  Initiative 


Instructions 


This  survey  is  part  of  a  large,  on-going  research  program  involving  the  U.S.  Air  Force,  a  number 
of  firms  in  the  defense  aerospace  sector,  and  academia.  The  research  program  is  investigating  the 
application  of  “lean”  management  and  production  practices  to  the  defense  aerospace  industry.  This 
survey  is  designed  specifically  to  assess  the  impact  of  various  attributes  of  the  Department  of 
Defense  (DoD)  acquisition  environment  on  the  application  of  lean  management  practices  at  the 
program  level. 

Your  cooperation  is  vital  to  the  success  of  this  study!  Please  answer  the  following 
questions  as  they  apply  specifically  to  your  individual  program  or  project.  If  your 
program  is  part  of  a  larger  program  office,  do  not  answer  the  questions  as  they  might  apply  to  that 
larger  organization. 

Please  be  candid  and  honest  in  your  responses.  We  understand  that  you  may  have  concerns  about 
confidentiality.  Several  measures  will  be  taken  to  ensure  that  your  responses  will  remain 
confidential.  Nobody  but  the  researchers  named  at  the  end  of  the  survey  will  have  access  to  the 
information  requested  in  this  survey.  All  analysis  of  the  survey  data  will  be  presented  in  the  form 
of  aggregated  statistics.  No  individuals  or  individual  programs  will  be  identified  in  the  analysis  or 
reporting  of  the  responses.  We  understand  that  the  success  of  any  research  depends  upon  the 
quality  of  the  information  on  which  it  is  based,  and  we  take  seriously  our  responsibility  to  ensure 
that  any  information  you  entrust  to  us  will  be  protected  with  the  same  care  that  you  would  use 
yourself. 

This  survey  has  been  designed  in  2  sections.  Section  A  asks  questions  relating  to  program 
planning  and  execution,  and  should  require  no  more  than  60  minutes  to  complete,  on  average. 
Section  B  asks  more  detailed  questions  about  the  background  and  characteristics  of  the  program, 
and  should  take  no  more  than  90  minutes  to  complete,  on  average.  To  ensure  accuracy  and 
consistency  of  the  results,  we  would  prefer  that  the  program  manager  fill  out  the  entire  survey. 
However,  if  this  is  not  possible,  it  is  important  that  the  program  manager  at  least  attempt  to  fill  out 
section  A.  Please  ensure  that  both  sections  are  completed,  however.  Also  note  that  additional 
information  is  requested  at  the  end  of  the  survey  that  may  require  the  program  manager’s  attention. 

When  you  have  completed  the  survey,  please  return  it  to  the  point  of  contact  at  your  center  or 
command,  or  enclose  it  in  the  accompanying  envelope  and  return  it  directly  to  the  researchers.  We 
would  prefer  that  you  return  the  completed  survey  to  us  as  soon  as  possible,  but  to  ensure  timely 
and  complete  reporting,  please  return  it  by  February  23,  199  6.  If  you  would  like  feedback  on 
the  results  of  the  survey,  place  a  check  mark  in  the  space  provided  at  the  end  of  the  survey.  Thank 
you  for  your  help  in  this  important  project. 


Program  Name: 

Project  Name  (if  applicable): 
Development/Logistics  Center: 


Section  A:  Program  Management 


A .  1  What  is  the  DoD  acquisition  category  (ACAT)  designation  for  this  program? 


A .  2  In  what  DoD  acquisition  phase  is  your  program  currently? 


□  Pre-milestone  0 

□  Concept  Exploration  &  Definition 

□  Demonstration  &  Validation 

□  Engineering  &  Manufacturing  Development 


□  Production  &  Deployment 

□  Operation  &  Support 

□  Close-out/Cancellation 

□  Other: _ 


A .  3  Does  your  current  program  have  its  origins  in  another  program? 

□  No.  □  Yes. 


A .4  Is  the  program  office  that  is  responsible  for  administering  this  program  a  “basket”  or  multi- 
product  program  office  (responsible  for  the  administration  of  several  related  programs,  in 
various  stages  of  the  acquisition  cycle)? 

□  No.  □  Yes. 


A  .5  Does  the  funding  and/or  management  of  this  program  involve  more  than  one  branch  of  the 
U.S.  armed  forces  or  government  agencies? 

□  No. 

□  Yes 

->  Please  indicate  your  partner(s) _ 

->  Please  indicate  the  lead  partner _ 


A .  6  Which  of  the  following  groups  or  organizations  originally  promoted  the  technical/operating 
concept  and/or  provided  the  initial  funding  that  began  your  program? 


□  Eventual  user  or  focal  point  command 

□  Logistics  community 

□  Service  development  center/laboratory 

□  Own  service  headquarters 

□  Another  military  service 


□  (D)ARPA 

□  Elected  or  appointed  government  official 

□  Contractor/Industry 

□  Other _ 


A. 7  Does  this  program  involve  international  cooperative  agreements  or  foreign  military  sales 
(FMS)? 

□  No.  □  International  cooperation  or  partnerships  planned. 

□  FMS  planned.  □  International  cooperation  or  partnerships  currently 

□  FMS  currently  underway.  underway. 


A. 8  How  many  other  existing  programs  do  you  know  of  that  are  competing  to  fulfill  the  same 
or  similar  mission  objectives  as  your  program? 

Within  your  branch  of  the  U.S.  armed  services  _ 

Within  another  branch  of  the  U.S.  armed  services  - 

Programs  outside  of  the  U.S.  - 

A .  9  What  is  your  rank  and  title  or  position? 

Rank  or  civil  service  grade:  _ _ _ — 

Title  or  position:  .  _ _ _ 


A .  1 0  How  long  have  you  been  the  program  manager  for  this  program? 
_ (years) _ (months) 


A. 11  How  many  different  government  program  managers  have  led  this  program  since  milestone 
I  (or  program  initiation)? _ 


A.  12  How  many  programs  involving  technical  or  mission  objectives  similar  to  this  program’s 
have  you  managed  prior  to  working  on  this  specific  program? - 


A.  13  Please  indicate  the  courses  or  training  (lasting  at  least  one  week)  that  you  have  taken  or 
completed: 


□  Sys  1 00  or  equivalent 

□  Sys  200  or  equivalent 

□  Sys  400  or  equivalent 

□  DoD  Systems  Engineering 
Management  Course 


□  DSMC  Program  Management  Course  (PMC) 

□  Product  Development  at  a  business  school  or 
university 

□  Other(s)  _ 


A.  14  Please  rank  the  order  of  importance  of  the  following  considerations  in  meeting  your 
program’s  objectives  (“1”  indicates  the  item  with  the  greatest  importance  in  meeting  your 
program’s  objectives,  “4”  indicates  the  item  with  the  least  importance  in  meeting  your 
program’s  objectives): 

_  Low  acquisition  cost. 

_ Low  operational  and  support  cost. 

_  Shortened  schedule  to  reach  operational  capability. 

_  Superior  technical  performance  over  existing  systems. 


A.  15  To  what  extent  have  the  following  items  associated  with  your  program  changed  since 


milestone  I  (the  program’s  start)? 

No  Chance 

Some  Change 

Entirely  New 

Performance  requirements  1  2 

Mission  concept  l  2 

Threat/Adversary  l  2 

3  4  5 

3  4  5 

3  4  5 

6  7 

6  7 

6  7 

A.  16  To  what  extent  would  you  say  that  your  program’s  progress  in  meeting  its  overall 
objectives  has  been  effected  by  program  instability  (unplanned  changes  in  your  requested 
annual  budget  or  other  critical  resources)? 

No  Effect  Some  Effect  Major  Effect 

1  2  3  4  5  6  7 


A.  17  Please  estimate  your  program’s  cost  growth  to  date.  Calculate  this  as  the  percentage 
change  in  actual  expenditures  compared  with  initially-planned  expenditures  for  this  point  in 
your  program’s  progress,  using  actual-year  dollars.  Treat  increases  beyond  original  plan 
as  positive  cost  growth. 

Total  program  cost  growth  _ (+/-%  to-date)  (indicate  baseline  year) _ 

Now,  estimate  the  percentage  of  your  program’s  cost  growth  to  date  (calculated  above) 
that  can  be  directly  attributable  to  unexpected  changes  in  your  requested  annual  funding 
levels  (funding  instability),  changes  in  the  program’s  technical  requirements,  technical 
problems,  or  other  reasons,  respectively. 

%  of  cost  growth  due  to  funding  instability  _  _ (%) 

%  of  cost  growth  due  to  requirements  changes _  _ (%) 

%  of  cost  growth  due  to  technical  problems  _  _ (%) 

%  of  cost  growth  due  to  other  reasons  (%)  (please  specify:) _ 

100%  _ 


A.  18  To  what  extent  do  you  think  the  following  factors  have  introduced  instability  into  your 
program’s  planning  and  execution? 


Changes  in  annual  budget  allocations 
or  production  rates. 

Not  a 
Factor 

i 

2 

3 

Somewhat 

Factor 

4 

a 

5 

The  Primary 
Factor 

6  7 

Changes  in  user  requirements. 

i 

2 

3 

4 

5 

6 

7 

Unanticipated  technical  challenges 

originating  within  this  program. 

i 

2 

3 

4 

5 

6 

7 

Problems  resulting  from  technical 

challenges  in  other,  related  programs. 

i 

2 

3 

4 

5 

6 

7 

Problems  relating  to  cooperation  with 
another  service,  agency,  or 
government 

i 

2 

3 

4 

5 

6 

7 

Program  staffing  (government  or 

industry)  changes. 

1 

2 

3 

4 

5 

6 

7 

Problems  resulting  from  poor 

contractor  or  supplier  performance. 

1 

2 

3 

4 

5 

6 

7 

Changing  DoD  or  Service  acquisition 

priorities. 

i 

2 

3 

4 

5 

6 

7 

Long  acquisition  cycle  time 

i 

2 

3 

4 

5 

6 

7 

Other: 

i 

2 

3 

4 

5 

6 

7 

A.  19  By  what  percentage  can  your  annual  budget  vary  before  a  significant  amount  of  effort 
(beyond  that  typically  required  for  normal  program  control  and  execution)  is  required  to  re¬ 
plan  the  program’s  work  schedule  or  objectives? 

_ .(%) 


A.  20  Please  estimate  the  amount  of  time  in  your  program  (in  annual  full-time  equivalent  staff  or 
“man-years”)  that  is  spent  exclusively  planning  your  program’s  schedule^  and  execution 
in  anticipation  of  or  as  a  result  of  unforeseen  changes  in  funding  or  other 
key  resources: 

Planning  prior  to  annual  appropriation  (“budget  exercises”)  _ (FTE) 

Re-planning  in  response  to  actual  budget  changes  _ (FTE) 


A.21  Please  indicate  the  extent  to  which  you  believe  the  following  practices  have  had  an  impact 
in  helping  to  assure  stability  (i.e.,  avoid  instability)  in  your  program’s  funding  and/or 


requirements: 

Not 

Used 


User  involved  in  developing 
requirements  0 

Contractor  involved  in  developing 
requirements  0 

Independent  experts  assess  program 
plan  and  execution  0 

Contractor  involved  in  developing 
initial  program  schedule  0 

Short  overall  acquisition  time  0 

Design  based  on  incremental 
technology  steps  0 

High-risk  development  items 

managed  off  the  critical  program 

path  0 

Interdependent  technical  activities 

scheduled  only  within  the  same 

budget  years  0 

Designs  validated  using  prototypes  0 

Designs  validated  using  simulation 

and  modeling  0 

Multi-year  procurement  0 

Contract  flexibility  (e.g.,  priced 
options,  umbrella  contracts)  0 

Aggressively  advocating  support  for 
your  program  to  senior  acquisition 
officials  in  your  service,  DoD,  or 
Congress  0 


Not 

Significant 

Somewhat 

Significant 

Very 

Significant 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

1 

2 

3 

4 

5 

6 

7 

A. 2 2  Please  estimate  the  priority  to  assure  stable  funding  at  the  requested  budget  levels  for  this 
program  at  each  of  die  following  points  in  the  acquisition  hierarchy: 


Very  Lowest  Average 

Priority  Priority 


Very  Highest 
Priority 


User  l 

Development/Logistics  Center/PEO  l 
Service  l 

DoD  l 

Administration  1 

Congress  l 


2  3 
2  3 
2  3 
2  3 
2  3 
2  3 


4  5 
4  5 
4  5 
4  5 
4  5 
4  5 


6  7 
6  7 
6  7 
6  7 
6  7 
6  7 


A. 2 3  Please  indicate  the  extent  to  which  you  have  found  the  following  management  practices 
effective  at  mitigating  the  negative  impact  of  unplanned  budget  cuts  to  your  program  after 
they  have  occurred: 

Not 
Used 


Management  reserve  in  the  program 


Unused  funding  from  prior  years  0 

Using  temporary  (TEMS)  personnel 

for  staffing  0 

Using  personnel  from  other  offices  or 
laboratories  on  an  as-needed  basis  0 

Using  Computer-aided  scheduling 

and  task  breakdown  planning  0 

Using  3-D  modeling  for  design  0 

Using  computer-aided  manufacturing  o 

Using  flexible  assembly  techniques  0 

Involving  User  community  in 

program  decision-making  0 

Involving  Oversight  community  in 
program  decision-making  0 

Structuring  work  around  integrated 
product  teams  (IPTs)  0 

Managing  all  major  subsystems 

within  a  single  program  office  0 

Managing  major  program  subsystems 
in  separate  program  offices  0 


Not 

Somewhat 

Very 

Effective 

Effective 

Effective 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

1  2 

3 

4 

5 

6 

7 

A  .24  To  what  extent  do  you  agree  or  disagree  with  the  following  statement: 

“The  technologies  used  in  this  program  have  more  demanding 
requirements  than  commercial  technologies  and  therefore  this  system 
should  take  longer  to  develop  and  field .  ” 


Strongly 

Somewhat 

Strongly 

Disagree 

Apree 

Agree 

1  2 

3  4  5 

6  7 

A. 2 5  Is  your  program  considered  “fast  track”  (the  schedule  has  officially  been  accelerated  so  as 
to  shorten  the  total  acquisition  cycle)? 

□  No.  □  Yes. 

A. 2 6  Please  estimate  your  program’s  schedule  slip.  Calculate  this  as  the  difference  between  your 
current  schedule  and  the  initial  program  schedule  for  meeting  specific  milestones  at  this 
point  in  your  program  (treat  increases  beyond  original  plan  as  positive  schedule  slip). 

Total  program  schedule  slip  _ (+/-  months  to-date) 

Now,  estimate  the  percentage  of  your  program’s  schedule  slip  to  date  (calculated 
above)  that  can  be  directly  attributable  to  unexpected  changes  in  your  requested  annual 
funding  levels  (funding  instability),  changes  in  the  program’s  technical  requirements, 
technical  problems,  or  other  reasons,  respectively. 

%  of  schedule  slip  due  to  funding  instability 
%  of  schedule  slip  due  to  requirements  changes 
%  of  schedule  slip  due  to  technical  problems 
%  of  schedule  slip  due  to  other  reasons 


A. 2 7  In  your  opinion,  how  much  time  does/did  the  following  actions  add  to  your  critical 
schedule  path  in  months  from  program  initiation  (milestone  0)  to  delivery  of  the  first 
production  item?  The  critical  schedule  path  is  the  sum  of  all  non-overlapping  activities  that 
uniquely  contribute  to  the  overall  schedule  length.  The  sum  of  these  items  should  equal  the 
total  development  time. 


Total  Development  Time  _  Months 

Requirements  Process  _  Months 

Technology  Development  _  Months 

Engineering  Design.  Months 

Manufacturing  Design  _  Months 

Testing  Programs  Months 

Support  Development  Months 

Production  Months 

Bureaucratic  Delays  Months 

Other _  Months 


_ (%) 

_ (%) 

_ (%) 

_ (%)  (please  specify:) 

100%  _ 


A. 2 8  Please  rank  the  following  items  in  terms  of  their  importance  to  your  personal  performance 
rating  and  your  potential  for  promotion  (“1”  indicates  the  item  with  the  greatest  importance 
to  your  potential  for  promotion,  “4”  indicates  the  item  with  the  least  importance  to  your 
potential  for  promotion): 

_  Completing  program  under  budget 

_  Completing  program  ahead  of  schedule 

_  Exceeding  system  technical  performance  requirements 

_  Exceeding  operability/maintainability  requirements 


A. 2 9  From  the  standpoint  of  a  program  manager,  how  difficult  would  it  be  to  re-plan  your 
program  schedule  to  lengthen  your  current  schedule  by  20%  ? 

Not  Somewhat  Very 

Difficult  Difficult  Difficult 

1  2  3  4  5  6  7 


A. 30  From  the  standpoint  of  a  program  manager,  how  difficult  would  it  be  to  re-plan  your 
program  schedule  to  shorten  your  current  schedule  20%  ? 


Not 

Somewhat 

Very 

Difficult 

Difficult 

Difficult 

1  2 

3  4  5 

6  7 

A. 3 1  Assume  your  contractor  submitted  a  proposal  to  significantly  shorten  the  total  development 
time  on  your  program  at  no  additional  cost  to  the  government,  but  which  did  require  a 
significant  change  in  the  program’s  funding  profile  in  the  out-years.  How  long  do  you 
estimate  it  would  take  to  either  formally  accept  or  reject  the  proposal? _ (months) 


A .  3  2  During  war  time  a  number  of  systems  have  had  their  developments  completed  early  in  order 
to  enter  into  service.  In  your  opinion,  if  your  program  were  deemed  absolutely  essential  to 
a  war  time  effort,  how  long  would  it  take  to  field  the  first  production  item  without  losing  a 
significant  degree  of  capability?  _ (months) 


A. 3 3  In  response  to  unforeseen  events,  to  what  extent  are  you  likely  to 

change  each  of  the 

following  in  order  to  achieve  your  overall  program  objectives? 

Least  Likely 

Somewhat  Likely 

Most  Likely 

to  Change 

to  Chance 

to  Chance 

Total  program  cost 

1  2 

3 

4  5  6 

7 

Total  schedule  length 

1  2 

3 

4  5  6 

7 

System  performance 

1  2 

3 

4  5  6 

7 

Please  note  that 

additional  information  is 

requested  on  the  last  page  of 

this  survey 

booklet  that  you 

may 

be  required  to 

complete. 

Section  B:  Program  Control 


B.l  How  many  people  currently  work  on  the  government  side  of  this  program  (include  all 
people  working  with  the  government  program  office  such  as  TEMS  and  other  contract 
support)? 

Full-time  _ 

Part-time  .  . 

Total  full-time  equivalents  (h  i  t)  _ 


B.2  Place  a  check  mark  in  any  of  the  boxes  where  the  program  phases  in  the  rows  are 
concurrent  (overlap)  with  those  in  the  columns,  according  to  your  program  execution  plan: 


Acquisition  Phase 

CE 

EMD  FRP 

Concept  Exploration  (CE) 

^  -  4.  „  ^  W«*»£V -I 

It 

Demonstration/Validation  (DEM/VAL) 

Prototype  Development 

1 _ 

Engineering  &  Manufacturing  Development  (EMD) 

Operational  Test  &  Evaluation  (OT&E) 

Low  Rate  Initial  Production  (LRIP) 

Full-rate  Production  (FRP) 

B .  3  How  many  different  funding  sources  are  there  for  this  program? 

Funding  organization(s)  _ 

Program  elements  (“colors  of  money”)  _ 


B .  4  What  is  your  program’s  current  total  planned  or  actual  budget  expenditure  for  all  program- 
related  expenses,  in  each  of  the  following  program  phases?  (use  actual  dollars  and  indicate 
the  reference  year) 


Concept  Evaluation 

_ (Smillion) 

FY(s) 

Planned  /  Actual  (circle  one) 

DEM/VAL 

_ (Smillion) 

FY(s) 

Planned  /  Actual  (circle  one) 

HMD 

_ (Smillion) 

FY(s) 

Planned  /  Actual  (circle  one) 

Production  . . 

_ (Smillion) 

FY(s) 

Planned  /  Actual  (circle  one) 

B.5  What  is  the  current 

“fly-away”  unit  cost  of  your  program’s  product  (exclude  spares, 

training,  and  support  costs)? 

_ ($) 

-» Please  estimate  the  percentage  of  the  total  fly-away  unit  cost  of  your  program’s  product 
that  is  represented  by  off-the-shelf  (OTS)  or  non-developmental  items  (NDI): 

_ (%) 


B .  6  How  many  units  or  systems  are  planned  to  be  procured  (the  “total  buy”)  over  the  life  of  this 
program,  and  how  has  this  change  impacted  unit  (“fly-away”)  cost,  if  at  all? 

Initial  plan  at  milestone  I  _ (total  number  of  units) 

Current  plan  _ (total  number  of  units) 

%  Change  in  unit  cost  _ (+/-%) 


B .  7  How  many  times  has  the  total  number  of  units  that  are  planned  to  be  procured  over  the  life 
of  this  program  been  changed? 


B .  8  Please  estimate  the  extent,  on  average,  to  which  your  budget  has  been  changed  from  your 
original  annual  submissions  at  each  of  the  following  levels  of  the  acquisition  hierarchy: 

User  _ (+/-%  this  year)  _  (+/-%  on  average) 

Development/Logistics  Center/PEO  _  (+/-  %  this  year)  _  (+/-  %  on  average) 

Service  _  (+/-  %  this  year)  _  (+/-  %  on  average) 

DoD  _  (+/-%  this  year)  _  (+/-%  on  average) 

Administration  _ (+/-  %  this  year)  _  (+/-  %  on  average) 

Congress  _ (+/-  %  this  year)  _  (+/-  %  on  average) 


B .  9  How  many  times  has  this  program  been  re-phased  (a  significantly  new  program  schedule 
developed)  since  the  milestone  I  decision  (or  program  initiation)? _ 

How  many  times  were  these  new  program  plans  created  in  each  program  phase? 

Concept  Exploration  phase  _ 

Demonstration/Validation  phase  _ 

Engineering  and  Manufacturing  Development  phase  _ 

Production  phase  _ 

Operation  and  Support  phase  _ _ 

Cancellation/Close-out  phase  _ _ 


B .  1 0  Please  provide  the  following  information  about  the  initial  planning  and  the  actual  and/or 
current  plan.  If  the  information  is  not  applicable  please  indicate  with  an  N/A.  Please 
provide  month  and  year  (8/93)  or  quarter  and  year.  Please  estimate  if  exact  dates  are 
unknown. 


N/A 

■jsnsum 

Actual/Current 

Mission  Needs  Statement 

Milestone  0 

Milestone  I 

Initial  Acquisition  Strategy  Plan 

Milestone  II 

Development  of  EMD  Program  Schedule 

Preliminary  Design  Review 

Critical  Design  Review 

Milestone  EH 

Completion  Of  First  Production  Item 

Initial  Operational  Capability 

Note:  N/A — Not  Applicable  or  no  date  available. 


B .  1 1  Please  select  the  category  below  that  best  describes  the  type  of  system  your  program  is 
developing: 


□  Aircraft  (airframe  and  mechanical  systems)  □ 

□  Aircraft  (avionics  and  electronic  systems)  □ 

□  Aircraft  (propulsion)  □ 

□  Spacecraft  or  launch  system  □ 


Electronic  system  (non-aircraft) 
Missile  or  munitions 
Software-dominated  system 
Other: _ 


B.12  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
product  technologies  required  by  your  program,  relative  to  existing  systems: 

i 

□  Revolutionary  new  core  technologies  or  concept. 

□  New  generation  of  product  architecture  or  platform. 

□  Incremental  improvements  to  an  existing  generation  of  product  architecture  or  platform. 

□  Little  or  no  change  to  the  existing  product  technology. 


B.13  Please  mark  the  response  that  best  describes  the  scope  of  the  technological  advance  in 
process  (manufacturing)  technologies  required  by  your  program,  relative  to  existing 
processes: 

□  Revolutionary  new  core  process  technology  or  concept. 

□  New  generation  of  existing  processes. 

□  Incremental  improvements  to  an  existing  generation  of  process  technologies. 

□  Little  or  no  change  to  the  existing  process  technology. 


B .  1 4  Please  select  the  response  which  best  describes  the  level  of  technological  advance  required 
in  the  component(s)  that  you  consider  to  be  critical  to  fulfilling  this  program’s  technical 
objectives: 


All  Non- 
Development 
Items 

1  2 


Some 

Development 

Ramim! 

4 


All  New 
Development 
Required 

6  7 


B .  1 5  Please  estimate  the  overall  extent  to  which  achieving  your  program  objectives  requires  the 
completion  of  interdependent  tasks  across  different  areas  of  your  program. 
Interdependencies  occur  when  developments  in  one  area  of  the  program  have  a  significant 
impact  on  developments  in  another,  and  vice  versa.  Consider  the  impact  of  activities 
involving  contractors,  suppliers,  users,  and  related  systems  being  managed  by  other 


program  offices  as  part  of  your  answer: 

Very  Little 

Some 

Very  Significant 

Interdependencv 

Interdependencies 

Interdependencies 

1  2 

3  4  5 

6  7 

B.16  Have  the  performance  capabilities  of  any  of  the  underlying  technologies  that  you  consider 
to  be  key  to  the  effectiveness  of  your  system  (e.g.,  digital  computer  processors,  advanced 
materials)  changed  significantly  during  the  duration  of  your  program,  exclusive  of  any 
technological  advances  made  by  this  program? 

No  Change  Some  Chanees  Major  Changes 

1  2  3  4  5  6  7 


B .  1 7  Please  estimate  the  percentage  of  the  total  cost  of  this  system  that  is  related  to  software: 
_ (%) 


B .  1 8  Please  estimate  the  percentage  of  the  staff  that  started  working  on  this  program  at  milestone 
I  (or  at  its  beginning)  who  are  still  working  on  this  program: 

_ .(%) 


B .  1 9  To  what  extent  is  a  short  acquisition  cycle  (quick  initial  operational  capability)  an  important 
part  of  this  program’s  overall  objectives? 

Not  Somewhat  Very 

Important  Important  Important 

1  2  3  4  5  6  7 

B .  2  0  Was  development  time  a  significant  evaluation  criteria  during  your  source  selection? 

Not  Somewhat  Very 

Important  Important  Important 

12  3  4  5  6  7 


B.21  Please  indicate  which  of  the  organizations  below  were  involved  in  the  following  activities. 
Also,  please  indicate  whether  or  not  the  organizations  indicated  met  regularly  as  part  of  an 
integrated  product  team  (EPT). 


SPO 

(V) 

User 

(V) 

Service 

(V) 

DoD 

(V) 

utm 

GAO 

(V) 

Hi 

n 

msm 

RttSl 

Requirements  Generation 

Acq.  Strategy  Development 

RFP  Generation 

Performance  Trade  Studies 

Funding  Profile  Development 

Program  Reviews 

Key: 

SPO— Program  office,  including  TEMS  and  contract  help  Prime— Potential  prime  contractors 

Service — Service  acquisition  community  Subs — Potential  subcontractors 

FFRDC — Mitre,  Aerospace  Corp.,  or  equivalent  IPT— Integrated  Product  Team 

GAO — General  accounting  office  or  other  congressional  or  governmental  oversight  agency 


B.22  Please  indicate  the  extent  to  which  the  following  factors  drove  the  length  of  the 
government’s  initial  system  development  schedule: 


No 

Impact 


Some 

Impact 


Defining 

Impact 


User’s  Desired  Schedule  1 

Service  Leadership  Desired  Schedule  l 

Service  Force  Planning  l 

Expected  Development  Funding  Levels  l 

Expected  Production  Funding  Availability  1 

Technology  Development  l 

Engineering  Requirements  l 

Manufacturing  Process  Development  l 

Testing  Requirements  1 

Support  Requirements  l 


Dependence  on  another  program’s  progress  l 


2  3 
2  3 
2  3 

2  3 
2  3 

2  3 
2  3 
2  3 
2  3 
2  3 
2  3 


4  5 
4  5 
4  5 

4  5 
4  5 

4  5 
4  5 
4  5 
4  5 
4  5 
4  5 


6  7 
6  7 
6  7 

6  7 
6  7 

6  7 
6  7 
6  7 
6  7 
6  7 
6  7 


B.23  Please  indicate  the  extent  to  which  the  following  tools  or  sources  of  information  were  used 
in  the  development  of  the  government’s  initial  schedule  for  your  program : 


Not 

Used 

Extensively 

Used 

Occasionally 

Used 

Historical  similar-program  data 

1 

2 

3 

4 

5 

6 

7 

Parametric  modeling  or  estimation 

1 

2 

3 

4 

5 

6 

7 

Expert  judgment 

1 

2 

3 

4 

5 

6 

7 

Bottom  up  schedule  development 

1 

2 

3 

4 

5 

6 

7 

Expected  funding  availability 

1 

2 

3 

4 

5 

6 

7 

Comparable  commercial  development 

1 

2 

3 

4 

5 

6 

7 

Contractor  proposed  schedule 

1 

2 

3 

4 

5 

6 

7 

B.24  Please  indicate  below  the  number  of  proposals  received  by  your  program  that  had 
schedules  that  differed  from  the  initial  government  schedule  estimate: 

_  Total  number  of  proposals  received. 

_  Number  which  exceeded  the  government  schedule  estimate  by  1 5%  or  more. 

_  Number  which  exceeded  the  government  schedule  estimate  by  5%  to  15%. 

_  Number  which  were  within  5%  of  the  government  schedule  estimate. 

_  Number  which  were  less  than  the  government  schedule  estimate  by  5%  to  15%. 

_ _ Number  which  were  less  than  the  government  schedule  estimate  by  15%  or  more. 


B .  2  5  What  scheduling  tools  are  used  in  your  initial  program  schedule  development? 


Not 

Used 

Extensively 

Used 

Occasionally 

Used 

Gantt 

1 

2 

3 

4 

5 

6 

7 

Milestone 

1 

2 

3 

4 

5 

6 

7 

Critical  Path  Management 

1 

2 

3 

4 

5 

6 

7 

PERT 

1 

2 

3 

4 

5 

6 

7 

System  center-based  scheduling  models 

1 

2 

3 

4 

5 

6 

7 

Other: 

1 

2 

3 

4 

5 

6 

7 

B.26  What  was  the  level  of  involvement  of  the  following  organizations  in  developing  the  initial 


schedule  for  your  program? 

Not 

Somewhat 

Extensively 

Involved 

Involved 

Involved 

Users 

Planning  shops  (XT) 

SPO  personnel 
Center  Headquarters 
FFRDC  (Mitre,  Aerospace,  etc.) 
PEO 

Service  Acquisition  Community 
Service  Budget  Community 
DoD  Acquisition  Community 
DoD  Budget  Community 
DoD  Planning  Community 
Potential  Contractors) 

Potential  Subcontractors 


1  2  3  4  5  6  7 
1  2  3  4  5  6  7 
1  2  3  4  5  6  7 
1  2  3  4  5  6  7 
12  3  4  5  6  7 
12  3  4  5  6  7 
1  2  3  4  5  6  7 
12  3  4  5  6  7 
1  2  3  4  5  6  7 
1  2  3  4  5  6  7 
12  3  4  5  6  7 
12  3  4  5  6  7 
1  2  3  4  5  6  7 


B.27  Please  indicate  the  extent  to  which  the  technology  or  system  being  developed  by  this 
program  will  require  the  following  changes  with  respect  to  the  user: 

No  Some  Entirely 

Change  Changes  No?! 

User  organization  l  2 

User’s  equipment  and  facilities  l  2 

User’s  operational  procedures  l  2 

User  that  controls  the  mission  l  2 


3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 
3  4  5  6  7 


B .  2  8  Please  select  the  response  that  best  describes  the  type  contract  that  accounts  for  the  greatest 
share  of  your  program’s  funding: 

□  Firm  Fixed  Price  (FFP)  □  Cost  Plus  Fixed  Fee  (CPFF) 

□  Fixed  Price  with  Economic  Price  Adjustment  □  Cost  and  Cost  Sharing  (C/CS) 

□  Fixed  Price  Incentive  (FPI)  □  Letter  Contract 

□  Fixed  Price  with  Redetermination  (FPR)  □  Indefinite  Deliveiy/Indefinite  Quantity 

□  Cost  Plus  Award  Fee  (CPAF)  □  Other - - - - 

□  Cost  Plus  Incentive  Fee  (CPIF) 


B.29  What  is  the  cost  share  ratio  at  which  cost  overruns  or  under-runs  near  the  target  price  are 
shared  between  government  and  contractor? _ %  Government  / - %  Contractor 

B  .30  What  percentage  of  your  contract  cost  is  available  as  an  incentive  for  on-time  completion  of 
a  major  milestone  or  activity? _ (%) 

B .  3 1  Exclusive  of  any  incentive  for  on-time  completion,  what  percentage  of  your  contract  cost  is 
available  as  an  incentive  for  early  completion  of  a  major  milestone  or  activity? 

_ (%) 


If  you  have  any  comments  or  recommendations  about  the  issues  addressed  by  this  survey,  feel  free 
to  write  them  in  the  space  provided  below: 


Please  note  that  additional  information  is  requested  on  the  last  page  of 
this  survey  booklet  that  you  may  be  required  to  complete. 


In  order  to  gain  an  understanding  of  the  industry  perspective  and  experience  on  these  important 
acquisition  issues,  we  would  like  to  send  a  similar  survey  to  your  counteipart  program  manager  at 
the  prime  contractors)  for  this  program.  Please  fill  in  his/her  name  and  address  information  in  the 
following  blanks.  You  will  not  be  identified  as  the  source  of  this  information,  nor  will  your 
responses  on  this  survey  be  reported  in  any  way  that  would  identify  you  to  your  industry 
counterpart. 


Name: 

Company: 
Mailing  Address: 


Name: 

Company: 
Mailing  Address: 


□  Yes,  I  would  like  a  summary  of  the  results  of  this  research  sent  to  me. 

Name:  _ 

Mailing  Address:  _ 


Tel: 


If  you  have  any  questions  about  this  survey  or  the  objectives  of  this  research,  feel  free  to  contact 
either: 

Dr.  Eric  Rebentisch 
Lean  Aircraft  Initiative 
Massachusetts  Institute  of  Technology 
77  Massachusetts  Ave.,  Room  33-407 
Cambridge,  MA  02139 
tel.:  (617)  258-7773 
fax:  (617)  258-7845 
email:  erebenti@mit.edu 

For  more  information  about  the  Lean  Aircraft  Initiative,  you  may  contact  us  directly  or  visit  the  LAI 
web  page  at:  http://web.mit.edu/ctpid/www/lai/. 


Capt.  Ross  McNutt 
Lean  Aircraft  Initiative 
Massachusetts  Institute  of  Technology 
77  Massachusetts  Ave.,  Room  33-407 
Cambridge,  MA  02139 
tel.:  (617)  258-6594 
fax:  (617)258-4235 
email:  mcnuttrt@mit.edu 


Thank  You  Very 


Much!!!! 
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